dev-tools:s3_object_storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
dev-tools:s3_object_storage [2025/07/21 12:16] – created rekonasdev-tools:s3_object_storage [2025/10/15 16:37] (current) – [RKNS (Zarr V2) from Minio ZIP] fabricio
Line 26: Line 26:
 <code>mc ls strg1</code> <code>mc ls strg1</code>
  
 +== Failed to verify certificate ==
 +If you get the following error, you still need to add the certificate (see bottom of the page) to your system.
 +<code> mc: <ERROR> Unable to list folder. Get "https://s3.strg1.lan/": tls: failed to verify certificate: x509: certificate signed by unknown authority </code>
 +
 +For adding the certificate:
 +
 +1. write the certificate it to a file.
 +
 +2. move it to the path where certificates are stored on your OS.
 +For OpenSuse Leap 15.6:
 +<code>/etc/pki/trust/anchors/</code>
 +For Ubuntu it should be:
 +<code>/usr/local/share/ca-certificates/</code>
 +
 +3.Run the following command to update the certificates (same command on Ubuntu or OpenSuse)  <code>update-ca-certificates</code> .
 ===== Example ''credentials.json'': ===== ===== Example ''credentials.json'': =====
  
Line 228: Line 243:
 zarr.create_array(store=store, shape=(2,), dtype="float64") zarr.create_array(store=store, shape=(2,), dtype="float64")
  
 +</code>
 +
 +===== Example Using mlflow =====
 +This assumes that you have already set up a MLFLow server with a Minio artifact store.
 +In addition, the client has to also provide the credentials to the Minio store, as it directly sends it to s3, not via the mlflow.
 +You can do so by setting a few environment variables.
 +Note that the code below assumes you have a local file called 'ca.crt.cer', and that you have the credentials in an env file that is loaded via python-dotenv.
 +
 +<code python>
 +
 +print("\nLoading environment variables from .env file...")
 +load_dotenv()
 +
 +# --- Configuration: Bridge your .env to what MLflow/boto3 expects ---
 +os.environ["MLFLOW_TRACKING_URI"] = "https://mlflow.basel.lan"
 +os.environ["MLFLOW_S3_ENDPOINT_URL"] = "https://s3.strg1.lan"
 +os.environ["AWS_ACCESS_KEY_ID"] = os.getenv("STORAGE_ACCESS_KEY")
 +os.environ["AWS_SECRET_ACCESS_KEY"] = os.getenv("STORAGE_SECRET_KEY")
 +
 +ca_cert_path = "ca.crt.cer"
 +if not os.path.exists(ca_cert_path):
 +    raise FileNotFoundError(f"Certificate file not found at: {ca_cert_path}")
 +os.environ["AWS_CA_BUNDLE"] = ca_cert_path
 +# uncomment this if you want to ignore an insecure TLS/unsigned certificate
 +#os.environ["MLFLOW_TRACKING_INSECURE_TLS"] = "true"
 +os.environ["MLFLOW_TRACKING_CLIENT_CERT_PATH"] = ca_cert_path
 </code> </code>
  
Line 272: Line 313:
  
 </code> </code>
 +
 +
  
  • dev-tools/s3_object_storage.1753100176.txt.gz
  • Last modified: 2025/07/21 12:16
  • by rekonas