Verify locally first | Geo-MLOps Manual

Before uploading to the platform, you can check on your own PC that the image keeps the contract. All you need is Docker and one local MLflow server. The commands below are exactly what was run in the MNIST example directory, and the local MLflow is also started with the mlflow inside the example image (no Python packages need to be installed on the host).

1. Build the image

docker build -t example/mnist-trainer:v1 .

2. Create verification data

This imitates what the platform does for /geo/dataset. --synthetic creates synthetic images with digits drawn in, without a network.

docker run --rm -u "$(id -u):$(id -g)" -v "$PWD:/w" -w /w \
  --entrypoint python example/mnist-trainer:v1 \
  make_sample_data.py ./sample-data --synthetic --count 400
# -> sample-data/manifest.json (400 files)

3. Start a local MLflow

docker run -d --name mlflow-local --network host -u "$(id -u):$(id -g)" -e HOME=/tmp \
  -v "$PWD:/w" -w /w --entrypoint mlflow example/mnist-trainer:v1 \
  server --host 127.0.0.1 --port 5500 \
  --backend-store-uri sqlite:////w/store.db --artifacts-destination /w/mlartifacts

4. Create the run first, like the platform, and run training

The platform creates the run in advance and passes it in as MLFLOW_RUN_ID. Do the same so the verification conditions match the real thing.

export MLFLOW_TRACKING_URI=http://127.0.0.1:5500

RUN_ID=$(docker run --rm --network host -e MLFLOW_TRACKING_URI \
  --entrypoint python example/mnist-trainer:v1 -c "
from mlflow.tracking import MlflowClient
c = MlflowClient()
exp = c.get_experiment_by_name('local-verify')
exp_id = exp.experiment_id if exp else c.create_experiment('local-verify')
print(c.create_run(exp_id).info.run_id)" | tail -1)

docker run --rm --network host \
  -v "$PWD/sample-data:/geo/dataset:ro" \
  -e GEO_DATA_DIR=/geo/dataset \
  -e GEO_WORK_DIR=/geo/work \
  -e GEO_HP_EPOCHS=2 \
  -e GEO_PARAM_DEVICE=cpu \
  -e MLFLOW_TRACKING_URI \
  -e MLFLOW_EXPERIMENT_NAME=local-verify \
  -e MLFLOW_RUN_ID=$RUN_ID \
  example/mnist-trainer:v1
echo "exit=$?"

-> train 320 / val 80 @ cpu
-> epoch 1/2 loss=2.2821 val_acc=0.1000
-> epoch 2/2 loss=2.1825 val_acc=0.3500
-> done. best val accuracy = 0.3500
🏃 View run intrigued-hen-480 at: http://127.0.0.1:5500/#/experiments/1/runs/79676e08…
exit=0

--network host is there because MLflow is on localhost of the same PC. You do not need it if you use a remote MLflow address.

5. What to check

#	Check	If it fails
1	Is the exit code `0`?	The run is recorded as failed even if all metrics are there
2	Are metric curves drawn in that MLflow run (`$RUN_ID`)?	The training screen stays empty while it runs. Check that you did not overwrite `MLFLOW_RUN_ID` or call `set_experiment()`
3	Are the param `epochs` and the metric `step` present?	It shows only "Training (학습 중)" with no progress bar
4	Is `model` logged in the run, with `MLmodel` · `signature` visible?	Registry registration is skipped and the model cannot go to serving
5	Are the outputs you need kept in MLflow?	`/geo/work` is not collected
6	Does the serve check below pass?	Training works but the model cannot be deployed — this is the most important item

Open the MLflow screen (http://127.0.0.1:5500) to check 1–5 by eye.

6. Serve check — a precondition for deployment

The platform's inference serving starts a registered model version as is with the MLflow scoring server. If it starts locally, it starts on the platform too.

# 1) Register the model logged by training in the local registry (automatic on the platform)
docker run --rm --network host -e MLFLOW_TRACKING_URI --entrypoint python \
  example/mnist-trainer:v1 -c "
import mlflow
from mlflow.tracking import MlflowClient
c = MlflowClient()
run = c.get_run('$RUN_ID')
m = c.search_logged_models(experiment_ids=[run.info.experiment_id],
                           filter_string=\"source_run_id = '$RUN_ID'\")[0]
mv = mlflow.register_model(f'models:/{m.model_id}', 'mnist')
print('registered', mv.name, mv.version)"

# 2) Start the registered version as is
docker run -d --name mnist-serve --network host -e MLFLOW_TRACKING_URI \
  --entrypoint mlflow example/mnist-trainer:v1 \
  models serve -m models:/mnist/1 -h 127.0.0.1 -p 5501 --env-manager local

# 3) One request — do you get 200 and a JSON prediction?
python3 -c "
import base64, json
b = base64.b64encode(open('sample-data/images/00001.png','rb').read()).decode()
json.dump({'dataframe_split': {'columns': ['image_b64'], 'data': [[b]]}}, open('input.json','w'))"
curl -X POST 127.0.0.1:5501/invocations -H 'Content-Type: application/json' -d @input.json

{"predictions": [{"label": "1", "confidence": 0.11639321595430374}]}

What matters here is whether a response comes back (whether the model loads and accepts requests according to the signature). Synthetic data gives a weak training signal, so a wrong prediction is fine.

Do the final check with --env-manager virtualenv

The example above starts the server inside the training image, so it uses --env-manager local (the current environment as is). This mode does not install pip_requirements, so it cannot catch version typos. For the last check before handing over, use --env-manager virtualenv, which actually builds an isolated environment from pip_requirements (requires pyenv and virtualenv).

mlflow models serve -m "models:/mnist/1" -p 5501 --env-manager virtualenv

The three most common failures — missing signature, predictor.py importing training code, and pip_requirements differing from the training environment — show no symptoms during training and surface only in serving. That is why you must catch them here.

Clean up

docker rm -f mlflow-local mnist-serve

Next: Upload the image to the platform

1. Build the image#

2. Create verification data#

3. Start a local MLflow#

4. Create the run first, like the platform, and run training#

5. What to check#

6. Serve check — a precondition for deployment#

Clean up#

1. Build the image

2. Create verification data

3. Start a local MLflow

4. Create the run first, like the platform, and run training

5. What to check

6. Serve check — a precondition for deployment

Clean up