Context
I have containerised a machine learning inference batch job that I am currently running on an on-prem ubuntu instance as a cron job. Basically, the cron job runs (there are a few other lines, but this is the core)
docker run $IMAGE
and this performs some inference using python.
Challenge
I want to migrate this job to the cloud using GCP infrastructure, but a bit confused about which service to use and how. Someone from Google customer team initially suggested Google Batch, but after some research I realised batch cannot support cron job at all. Then some forums/tutorials seem to support Google cloud scheduler. But after further reading it seems they trigger jobs if I hit an http endpoint. (That means hitting the http endpoint has to be established as a cron job with my on-prem infra? But why would anyone use that?)
So want to ask the SOF community on what's the appropriate Google service to run a cron job, without myself managing an instance (and keeping it on 24*7).
For information, I have already pushed the $IMAGE
to Google artefact registry. So all I need for google is to run the image with the following cron expression.
30 0 * * *
Further Details (Based on Comment)
I have a docker image in google artefact registry. The last line of the image is
# Run the script at container boot time.CMD ["./run_manager.sh"]
This run_manager.sh
performs the feature engineering, model training+inference using tensorflow and persists the result in an appropriate database. The whole thing takes about 20-25 minutes to run and then the container terminates.I need to run this job daily, let's say at 12.30 am every day. Currently I have this set up as a cron job on an on premise Ubuntu instance.
Requirement
Leverage Google cloud to perform this, without having to keep a VM on 24*7 just for one job. The behaviour I want is a Google cloud service to which I supply
- The artefact URL in the artefact registry
- The cron expression (or something equivalent)
and then GCP takes care of running the artefact/container according to the cron expression?
There is no web-service, no http-endpoint, no pub-sub here. Just a cron job.