Just for the record, in a corporate manner, I ask ChatGPT to explain to me more precisely the meaning. Here is the explanation:<\/p>\n
The title “IA: Moving From POC To Scaling” describes a phase in AI project development where an AI solution that has successfully passed the Proof of Concept (POC) stage is now being prepared for scaling, meaning it will be expanded and deployed on a larger scale for widespread use. The transition requires careful planning and execution to ensure that the AI system can operate efficiently at a larger scale. This transition from POC to scaling is often seen as a critical milestone in AI development, where a project moves from testing feasibility to real-world implementation and growth.”<\/p><\/blockquote>\n
Concretely, this doesn’t say much about what should be done… But I tried to understand how to take advantage of the latest “using Prompt Engineering” feature of MLflow.<\/p>\n
Why that because, this MLflow’s feature allows you to track LLM responses across different settings of model temperature, max output tokens and of course prompts. This allows for transparent prompt engineering and parameter tuning… And above all you can also leverage on self-hosted LLMs on Ollama!<\/p>\n
Mostly, I use Ollama to operate open-source LLMs to have extensive control over the confidentiality of the content sent to the LLMs and to reduce drastically the expenses.<\/p>\n
I also ended up finding a video and plenty of documentation from Mlflow.<\/p>\n
\n
- A very educational notebook that fully describes the process of “Comparing LLMs with MLFlow”.<\/a> Like it is said by the author “This notebook demonstrates how to use MLFlow to compare different text generation models from Hugging Face and compare different generation configurations for those models.”<\/li>\n
- MLflow: serving LLMs and prompt engineering<\/a><\/li>\n
- MLflow\u2019s Support for LLMs<\/a>. Nothing less than “aims to alleviate these challenges by introducing a suite of features and tools designed with the end-user in mind”<\/li>\n<\/ul>\n
Concretely, to make MLflow work with LLMs, it is necessary to complete the “Served LLM model” of MLflow to test indifferently open source or paid LLMs by declaring them within MLflow.<\/p>\n
The the main obstacle was to add “item” in “Served LLM model” in MLflow dropdown list.<\/i><\/p>\n
Screen captures from Daniel Liden djliden that show how to leverage on the MLflow’s “using Prompt Engineering” feature<\/b>. Check https:\/\/github.com\/djliden\/llmops-examples\/tree\/main<\/a><\/p>\n
Screen capture from Daniel Liden djliden<\/i><\/p>\n
Screen capture from Daniel Liden djliden<\/i><\/p>\n
Screen capture from Daniel Liden djliden<\/i><\/p>\n
Screen capture from Daniel Liden djliden<\/i><\/p>\n\r\n# mlflow_prompt_eng_ui_assets\/config.yaml\r\n# https:\/\/github.com\/djliden\/llmops-examples\/blob\/00b42c7ec0f7e5914bf77966e84ddbfe02230e18\/mlflow_prompt_eng_ui_assets\/config.yaml\r\n# https:\/\/mlflow.org\/docs\/latest\/llms\/gateway\/migration.html\r\n# https:\/\/github.com\/minkj1992\/llama3-langchain-mlflow\/blob\/main\/mlflow\/config.yaml\r\n\r\nendpoints: # Renamed to \"endpoints\"\r\n - name: chat\r\n endpoint_type: llm\/v1\/chat # Renamed to \"endpoint_type\"\r\n model:\r\n provider: openai\r\n name: gpt-3.5-turbo\r\n config:\r\n openai_api_key: $OPENAI_API_KEY\r\n - name: ollama\r\n endpoint_type: llm\/v1\/chat\r\n model:\r\n provider: openai\r\n name: llama3\r\n config:\r\n openai_api_key: \"\"\r\n # https:\/\/ollama.com\/blog\/openai-compatibility\r\n openai_api_base: http:\/\/host.docker.internal:11434\/v1\r\n<\/pre>\nAt the same time, it was necessary to familiarize oneself with how MLflow works. Indeed, given MLflow’s ability to create many experiments and runs, it is better to question a priori the tagging and search capacities and therefore the classification of MLflow to take full advantage of the organization induced by MLflow.<\/p>\n
Searching By Params<\/b><\/p>\n
\r\nparams.batch_size = \"2\"\r\nparams.model LIKE \"GPT%\"\r\nparams.model ILIKE \"gPt%\"\r\nparams.model LIKE \"GPT%\" AND params.batch_size = \"2\"\r\n<\/pre>\nSearching By Tags<\/b><\/p>\n
\r\ntags.\"environment\" = \"notebook\"\r\ntags.environment = \"notebook\"\r\ntags.task = \"Classification\"\r\ntags.task ILIKE \"classif%\"\r\n<\/pre>\n\r\nparams.model_route LIKE \"%mistral-ollama%\"\r\nparams.model_route LIKE \"%gpt4o-azure%\"\r\nparams.model_route LIKE \"%openhermes-ollama%\"\r\ntags.nid like \"%MZ344252%\"\r\n<\/pre>\nSource: https:\/\/mlflow.org\/docs\/latest\/search-runs.html<\/a><\/p>\n
A good introduction to MLFlow features<\/h2>\n
If you are looking for a complete and easy introduction to MLFlow on how it works in general. Here is a practical introduction to MLFlow to understand its essential concepts. Unfortunately, the “using Prompt Engineering” feature is not covered by this video series.<\/p>\n
This 32 videos’ playlist, made by Manuel Gil, illustrate the main concepts of MLflow <\/p>\n
https:\/\/www.youtube.com\/playlist?list=PLQqR_3C2fhUUkoXAcomOxcvfPwRn90U-g<\/a><\/p>\n
\r\n# A. INSTALL MLFLOW \r\n# 1. create a anaconda env named using_mlflow\r\n\r\n# Conda Environment\r\nconda create --name using_mlflow python=3.9.13\r\nconda info --envs\r\nsource activate using_mlflow\r\nconda deactivate\r\nsource activate using_mlflow\r\n\r\n# if needed to remove\r\nconda env remove -n [NAME_OF_THE_CONDA_ENVIRONMENT]\r\nconda env remove -n using_mlflow\r\n\r\n# 2. Install MLflow from PyPI using pip:\r\npip install mlflow\r\n\r\n# test the install\r\nmlflow --version\r\n\r\n# B. CREATING EXPERIMENTS IN MLFLOW\r\n# launch the UI\r\nmlflow ui\r\n\r\n# Check http:\/\/127.0.0.1:5000\r\n<\/pre>\nCollateral discoveries<\/h2>\n
1. Using Pydantic to control LLM output<\/b>
\nAlong the way, another area of improvement is the use of Pydantic to validate the generative AI JSON output format. Indeed, if you want to integrate this JSON response into an API or a webapp, it is better to ensure the validity and consistency of this response.<\/p>\nSource: https:\/\/medium.com\/@mattchinnock\/controlling-large-language-model-output-with-pydantic-74b2af5e79d1<\/a><\/p>\n
2. Using Crewai<\/b>
\nDuring this exploration, I discovered Crewai and also connect to my LLMs Ollama operated system.<\/p>\nSource: https:\/\/docs.crewai.com\/<\/a><\/p>\n
\r\n# for the examples available in github, it was required to downgrade crewai to 0.10.0\r\n# From crewai 0.51.1 to crewai 0.10.0\r\npip install crewai==0.10.0\r\n\r\n<\/pre>\n3. Anythingllm<\/b>
\nIt is worth mentioning this project “Anythingllm” which presents itself as a “all-in-one AI application that can do RAG, AI Agents, and much more with no code or infrastructure headaches”.<\/p>\nIt is possible to install it in three diverse ways in Desktop mode, via Docker or finally via Homebrew.<\/p>\n
Once installed in Desktop mode for example, which nevertheless requires 5GB of Storage, it is possible to connect to Ollama for instance locally via the Anythingllm settings and leverage on self-hosted llms provided by Ollama e.g phi3.5, openhermes, mistral-openorca, zephyr, orca-mini… etc <\/p>\n
Source: https:\/\/docs.anythingllm.com\/setup\/llm-configuration\/local\/ollama<\/a><\/p>\n
Source: https:\/\/anythingllm.com\/<\/a><\/p>\n
For model available, check https:\/\/anythingllm.com\/<\/a> <\/p>\n
More infos<\/H2><\/p>\n
MLflow<\/H3><\/p>\n
\n
- MLFlow: A Quickstart Guide – YouTube
https:\/\/www.youtube.com\/watch?v=cjeCAoW83_U<\/a><\/li>\n- 01. Introduction To MLflow | Track Your Machine Learning Experiments | MLOps – YouTube
https:\/\/www.youtube.com\/watch?v=ksYIVDue8ak<\/a><\/li>\n- MLflow for Machine Learning Development – YouTube
https:\/\/www.youtube.com\/playlist?list=PLQqR_3C2fhUUkoXAcomOxcvfPwRn90U-g<\/a><\/li>\n- GitHub – manuelgilm\/mlflow_for_ml_dev: Repository with code examples of mlflow
https:\/\/github.com\/manuelgilm\/mlflow_for_ml_dev<\/a><\/li>\n- Advancements in Open Source LLM Tooling, Including MLflow – YouTube
https:\/\/www.youtube.com\/watch?v=WpudXKAZQNI<\/a><\/li>\n- MLflow LLM Evaluate
https:\/\/mlflow.org\/docs\/latest\/llms\/llm-evaluate\/index.html<\/a><\/li>\n- mlflow\/examples\/evaluation at master \u00b7 mlflow\/mlflow \u00b7 GitHub
https:\/\/github.com\/mlflow\/mlflow\/tree\/master\/examples\/evaluation<\/a><\/li>\n- Announcing MLflow 2.4 for LLMOps | Databricks Blog
https:\/\/www.databricks.com\/blog\/announcing-mlflow-24-llmops-tools-robust-model-evaluation<\/a><\/li>\n- MLflow | ️ LangChain
https:\/\/python.langchain.com\/v0.2\/docs\/integrations\/providers\/mlflow_tracking\/<\/a><\/li>\n- GitHub – Netflix\/metaflow: :rocket: Build and manage real-life ML, AI, and data science projects with ease!
https:\/\/github.com\/Netflix\/metaflow<\/a><\/li>\n- llmops-examples\/compare-openai-transformers.ipynb at main \u00b7 djliden\/llmops-examples \u00b7 GitHub
https:\/\/github.com\/djliden\/llmops-examples\/blob\/main\/compare-openai-transformers.ipynb<\/a><\/li>\n- llmops-examples\/mlflow-prompt-eng-ui.ipynb at main \u00b7 djliden\/llmops-examples \u00b7 GitHub
https:\/\/github.com\/djliden\/llmops-examples\/blob\/main\/mlflow-prompt-eng-ui.ipynb<\/a><\/li>\n- LLM_Notebooks\/mlflow\/Deployment_Server\/mlflow_Serve.ipynb at 66d4b3a6d9d08813bb94ea653fd59275e66c91cc \u00b7 olonok69\/LLM_Notebooks \u00b7 GitHub
https:\/\/github.com\/olonok69\/LLM_Notebooks\/blob\/66d4b3a6d9d08813bb94ea653fd59275e66c91cc\/mlflow\/Deployment_Server\/mlflow_Serve.ipynb#L23<\/a><\/li>\n- GitHub – djliden\/llmops-examples: Example code and notebooks related to mlflow, llmops, etc.
https:\/\/github.com\/djliden\/llmops-examples\/tree\/main<\/a><\/li>\n- Comparing LLMs with MLFlow | Medium
https:\/\/medium.com\/@dliden\/comparing-llms-with-mlflow-1c69553718df<\/a><\/li>\n- LangChain within MLflow (Experimental)
https:\/\/mlflow.org\/docs\/latest\/llms\/langchain\/guide\/index.html<\/a><\/li>\n- Evaluate a Hugging Face LLM with mlflow.evaluate()
https:\/\/mlflow.org\/docs\/latest\/llms\/llm-evaluate\/notebooks\/huggingface-evaluation.html<\/a><\/li>\n- Prompt Engineering UI (Experimental)
https:\/\/mlflow.org\/docs\/latest\/llms\/prompt-engineering\/index.html<\/a><\/li>\n- Practical-Deep-Learning-at-Scale-with-MLFlow\/chapter01 at main \u00b7 PacktPublishing\/Practical-Deep-Learning-at-Scale-with-MLFlow \u00b7 GitHub
https:\/\/github.com\/PacktPublishing\/Practical-Deep-Learning-at-Scale-with-MLFlow\/tree\/main\/chapter01<\/a><\/li>\n- How to create a deep learning inference pipeline model using MLflow in three steps | by Yong Liu | Medium
https:\/\/medium.com\/@yong.liu_60428\/how-to-create-a-deep-learning-inference-pipeline-model-using-mlflow-in-three-steps-a567c534d751<\/a><\/li>\n- Comparing LLMs with MLFlow | Medium
https:\/\/medium.com\/@dliden\/comparing-llms-with-mlflow-1c69553718df<\/a><\/li>\n- MLflow Deployments for LLMs | ️ LangChain
https:\/\/python.langchain.com\/v0.2\/docs\/integrations\/providers\/mlflow\/<\/a><\/li>\n- Just a moment…
https:\/\/www.datacamp.com\/tutorial\/mlflow-streamline-machine-learning-workflow<\/a><\/li>\n- Exploring MLflow experiments with a powerful UI | by Gor Arakelyan | AimStack | Medium
https:\/\/medium.com\/aimstack\/exploring-mlflow-experiments-with-a-powerful-ui-238fa2acf89e<\/a><\/li>\n- Quickstart: Install MLflow, instrument code & view results in minutes \u2014 MLflow 2.7.0 documentation
https:\/\/mlflow.org\/docs\/2.7.0\/quickstart.html<\/a><\/li>\n- Quickstart: Compare runs, choose a model, and deploy it to a REST API \u2014 MLflow 2.7.0 documentation
https:\/\/mlflow.org\/docs\/2.7.0\/quickstart_mlops.html#quickstart-mlops<\/a><\/li>\n- Tutorials and Examples \u2014 MLflow 2.7.0 documentation
https:\/\/mlflow.org\/docs\/2.7.0\/tutorials-and-examples\/index.html#tutorials-and-examples<\/a><\/li>\n- Streamlining Text Classification Models with MLflow: A Comprehensive Guide | by Vasista Reddy | ScrapeHero | Medium
https:\/\/medium.com\/scrapehero\/streamlining-text-classification-models-with-mlflow-a-comprehensive-guide-6cc3ce71ed90<\/a><\/li>\n- GitHub – adamksiezyk\/data-science-workbench at mlflow-local-llm
https:\/\/github.com\/adamksiezyk\/data-science-workbench\/tree\/mlflow-local-llm<\/a><\/li>\n- Evaluating & Tracking LLMs using MLflow Model Evaluation & Phoenix -part-2 | by M K Pavan Kumar | . | Medium
https:\/\/medium.com\/aimonks\/evaluating-tracking-llms-using-mlflow-model-evaluation-phoenix-part-2-1830b3177abe<\/a><\/li>\n- Model Tracking with MLFlow & Deployment with FastAPI | Analytics Vidhya
https:\/\/medium.com\/analytics-vidhya\/fundamentals-of-mlops-part-4-tracking-with-mlflow-deployment-with-fastapi-61614115436<\/a><\/li>\n<\/ul>\nAnythingLLM<\/H3><\/p>\n
\n
- AnythingLLM | The all-in-one AI application for everyone
https:\/\/anythingllm.com\/<\/a><\/li>\n<\/ul>\ncrewAI<\/H3><\/p>\n
\n
- crewAI
https:\/\/docs.crewai.com\/<\/a><\/li>\n- crewAI – Platform for Multi AI Agents Systems
https:\/\/www.crewai.com\/<\/a><\/li>\n- GitHub – brooklynb7\/lang-ollama at 1c459019ab49414107a9f820cf1dd53750c3fa76
https:\/\/github.com\/brooklynb7\/lang-ollama\/tree\/1c459019ab49414107a9f820cf1dd53750c3fa76<\/a><\/li>\n- CrewAI Tutorial – Next Generation AI Agent Teams (Fully Local) – YouTube
https:\/\/www.youtube.com\/watch?v=tnejrr-0a94<\/a><\/li>\n- GitHub – crewAIInc\/crewAI: Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
https:\/\/github.com\/crewAIInc\/crewAI<\/a><\/li>\n- CrewAI Tutorial – Next Generation AI Agent Teams (Fully Local) – YouTube
https:\/\/www.youtube.com\/watch?v=tnejrr-0a94<\/a><\/li>\n- Search on “from crewai import Agent, Task, Crew ollama” on github
https:\/\/github.com\/search?q=from+crewai+import+Agent%2C+Task%2C+Crew+ollama&type=code<\/a><\/li>\n<\/ul>\nPydantic<\/H3><\/p>\n
\n
- Minimize LLM Hallucinations with Pydantic Validators | Pydantic
https:\/\/pydantic.dev\/articles\/llm-validation<\/a><\/li>\n- Enforce and Validate LLM Output with Pydantic | Timo’s Blog
https:\/\/timotk.github.io\/posts\/enforce-validate-llm-output-pydantic\/<\/a><\/li>\n- Pydantic and Prompt Engineering: The Essentials for Validating Large Language Model Outputs | by Aziz Ben Othman | Medium
https:\/\/medium.com\/@azizbenothman76\/pydantic-and-prompt-engineering-the-essentials-for-validating-language-model-outputs-e48553eb4a3b<\/a><\/li>\n- How to return structured data from a model | ️ LangChain
https:\/\/python.langchain.com\/v0.2\/docs\/how_to\/structured_output\/<\/a><\/li>\n- Tutorial Overview
https:\/\/mlflow.org\/docs\/latest\/getting-started\/logging-first-model\/index.html<\/a><\/li>\n- A Gentle Introduction to MLOps | Analytics Vidhya
https:\/\/medium.com\/analytics-vidhya\/fundamentals-of-mlops-part-1-a-gentle-introduction-to-mlops-1b184d2c32a8<\/a><\/li>\n- Data & Model Management with DVC | Analytics Vidhya
https:\/\/medium.com\/analytics-vidhya\/fundamentals-of-mlops-part-2-data-model-management-with-dvc-6be2ad284ec4<\/a><\/li>\n- ML Experimentation using PyCaret | Analytics Vidhya
https:\/\/medium.com\/analytics-vidhya\/fundamentals-of-mlops-part-3-ml-experimentation-using-pycaret-747f14e4c28d<\/a><\/li>\n- Model Tracking with MLFlow & Deployment with FastAPI | Analytics Vidhya
https:\/\/medium.com\/analytics-vidhya\/fundamentals-of-mlops-part-4-tracking-with-mlflow-deployment-with-fastapi-61614115436<\/a><\/li>\n<\/ul>\nOther<\/H3><\/p>\n
\n<\/p>\n
- GitHub – cric96\/langchain-examples: Basic examples of prompt engineering leveraging langchain: https:\/\/www.langchain.com\/
https:\/\/github.com\/cric96\/langchain-examples<\/a><\/li>\n- Deploy a local LLM | RAGFlow
https:\/\/ragflow.io\/docs\/dev\/deploy_local_llm<\/a><\/li>\n- teknium\/OpenHermes-2.5-Mistral-7B \u00b7 Hugging Face
https:\/\/huggingface.co\/teknium\/OpenHermes-2.5-Mistral-7B<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"I remain mostly an AI user and moreover without always understanding the subtleties of LMMs. This use does not prevent me from seeking to improve… <\/p>\n
![\"Enhance](\"https:\/\/flaven.fr\/wp-content\/uploads\/2024\/09\/tuto_mlflow_1.png\")
![\"Enhance](\"https:\/\/flaven.fr\/wp-content\/uploads\/2024\/09\/tuto_mlflow_2.png\")
![\"Enhance](\"https:\/\/flaven.fr\/wp-content\/uploads\/2024\/09\/tuto_mlflow_3.png\")
![\"Enhance](\"https:\/\/flaven.fr\/wp-content\/uploads\/2024\/09\/tuto_mlflow_4.png\")