Quantcast
Channel: Recent Questions - Stack Overflow
Viewing all articles
Browse latest Browse all 12111

How to Schedule a Cron Job on Google Cloud Scheduler?

$
0
0
Context

I have containerised a machine learning inference batch job that I am currently running on an on-prem ubuntu instance as a cron job. Basically, the cron job runs (there are a few other lines, but this is the core)

docker run $IMAGE

and this performs some inference using python.

Challenge

I want to migrate this job to the cloud using GCP infrastructure, but a bit confused about which service to use and how. Someone from Google customer team initially suggested Google Batch, but after some research I realised batch cannot support cron job at all. Then some forums/tutorials seem to support Google cloud scheduler. But after further reading it seems they trigger jobs if I hit an http endpoint. (That means hitting the http endpoint has to be established as a cron job with my on-prem infra? But why would anyone use that?)

So want to ask the SOF community on what's the appropriate Google service to run a cron job, without myself managing an instance (and keeping it on 24*7).

For information, I have already pushed the $IMAGE to Google artefact registry. So all I need for google is to run the image with the following cron expression.

30 0 * * *
Further Details (Based on Comment)

I have a docker image in google artefact registry. The last line of the image is

# Run the script at container boot time.CMD ["./run_manager.sh"]

This run_manager.sh performs the feature engineering, model training+inference using tensorflow and persists the result in an appropriate database. The whole thing takes about 20-25 minutes to run and then the container terminates.I need to run this job daily, let's say at 12.30 am every day. Currently I have this set up as a cron job on an on premise Ubuntu instance.

Requirement

Leverage Google cloud to perform this, without having to keep a VM on 24*7 just for one job. The behaviour I want is a Google cloud service to which I supply

  • The artefact URL in the artefact registry
  • The cron expression (or something equivalent)

and then GCP takes care of running the artefact/container according to the cron expression?

There is no web-service, no http-endpoint, no pub-sub here. Just a cron job.


Viewing all articles
Browse latest Browse all 12111

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>