nprasad2077/docker-docs
1
0 Fork 0
mirror of https://github.com/docker/docs.git synced 2026-11-04 10:01:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: remove low-use NLP and generic GenAI guides

Browse Source 
The NLP examples, TensorFlow.js face detection guide, Neo4j RAG guide,
and video chatbot guide have low readership and repeat application
setup or containerization covered by other guides.

Remove these eight guides and the nine images used only by them.
No remaining content or config entries reference the removed guides.
This commit is contained in:
David Karlsson committed 2026-10-08 13:39:03 +00:00
1 parent 8a308ef225
commit 7a1b2c0627
17 files changed
-2941
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

content/guides/genai-leveraging-rag.md
-242
View File
@@ -1,242 +0,0 @@
---
title: Leveraging RAG in GenAI to teach new information
linkTitle: Leveraging RAG in GenAI
description: This guide walks through the process of setting up and utilizing a GenAI stack with Retrieval-Augmented Generation (RAG) systems and graph databases. Learn how to integrate graph databases like Neo4j with AI models for more accurate, contextually-aware responses.
keywords: Docker, GenAI, Retrieval-Augmented Generation, RAG, Graph Databases, Neo4j, AI, LLM
summary: |
This guide explains setting up a GenAI stack with Retrieval-Augmented Generation (RAG) and Neo4j, covering key concepts, deployment steps, and a case study. It also includes troubleshooting tips for optimizing AI performance with real-time data.
params:
tags: [ai]
time: 35 minutes
---
## Introduction
Retrieval-Augmented Generation (RAG) is a powerful framework that enhances large language models (LLMs) by integrating information retrieval from external knowledge sources. This guide focuses on a specialized RAG implementation using graph databases like Neo4j, which excel in managing highly connected, relational data. Unlike traditional RAG setups with vector databases, combining RAG with graph databases offers better context-awareness and relationship-driven insights.
In this guide, you will:
* Explore the advantages of integrating graph databases into a RAG framework.
* Configure a GenAI stack with Docker, incorporating Neo4j and an AI model.
* Analyze a real-world case study that highlights the effectiveness of this approach for handling specialized queries.
## Understanding RAG
RAG is a hybrid framework that enhances the capabilities of large language models by integrating information retrieval. It combines three core components:
- **Information retrieval** from an external knowledge base
- **Large Language Model (LLM)** for generating responses
- **Vector embeddings** to enable semantic search
In a RAG system, vector embeddings are used to represent the semantic meaning of text in a way that a machine can understand and process. For instance, the words "dog" and "puppy" will have similar embeddings because they share similar meanings. By integrating these embeddings into the RAG framework, the system can combine the generative power of large language models with the ability to pull in highly relevant, contextually-aware data from external sources.
The system operates as follows:
1. Questions get turned into mathematical patterns that capture their meaning
2. These patterns help find matching information in a database
3. The LLM generates responses that blend the model's inherent knowledge with the this extra information.
To hold this vector information in an efficient manner, you need a special type of database.
## Introduction to Graph databases
Graph databases, such as Neo4j, are specifically designed for managing highly connected data. Unlike traditional relational databases, graph databases prioritize both the entities and the relationships between them, making them ideal for tasks where connections are as important as the data itself.
Graph databases stand out for their unique approach to data storage and querying. They use nodes (or vertices) to represent entities and edges to represent the relationships between these entities. This structure allows for efficient handling of highly connected data and complex queries, which are difficult to manage in traditional database systems.
SQL databases and graph databases differ significantly in their data models. SQL databases use a tabular structure with rows and columns, where relationships between entities are established using foreign keys. This approach works well for structured data and predefined relationships. In contrast, graph databases represent data as nodes (entities) and edges (relationships), making the representation of relationships more intuitive and flexible. This structure is particularly advantageous for complex, interconnected data.
Another key difference lies in schema flexibility. SQL databases operate on a rigid schema, meaning any changes to the data structure, such as adding new columns or altering relationships, typically require careful planning and migration processes. Graph databases, however, are schema-optional, allowing for much greater flexibility. New nodes, edges, or properties can be introduced without disrupting existing data, enabling faster adaptation to changing requirements.
## Practical implementation: testing RAG effectiveness
To illustrate the power of RAG systems in practice, let's examine a real-world implementation using Apache NiFi as our subject matter. This case study demonstrates how RAG can enhance an AI's ability to provide accurate, contextual information about specialized technical topics.
### Teaching AI about new technologies
Apache NiFi serves as an excellent example of the limitations of traditional LLMs and how RAG can overcome them. As a relatively recent technology, many LLMs have limited or outdated knowledge about it, making it a perfect candidate for demonstrating RAG's capabilities.
Without RAG, the LLM may provide broad responses or acknowledge its limited knowledge when asked about information that is beyond its cutoff date. The following example demonstrates this.
```text
Question: What important events happened in 2024?
RAG: Disabled
I'm happy to help! Unfortunately, I'm a large language model, I don't have access to real-time information or events that occurred after my training data cutoff in 2024. Therefore, I cannot provide you with any important events that happened in 2024. My apologize for any inconvenience this may cause. Is there anything else I can help you with?
```
## Setting up GenAI stack with GPU acceleration on Linux
To set up and run the GenAI stack on a Linux host, execute one of the following commands, either for GPU or CPU powered:
### GPU powered
```bash
git clone https://github.com/docker/genai-stack
docker compose --profile linux-gpu up -d
mv env.example .env
nano .env
```
In the `.env` file, make sure following lines are commented out. Set your own credentials for security
```txt
NEO4J_URI=neo4j://database:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=password
OLLAMA_BASE_URL=http://llm-gpu:11434
```
### CPU powered
```bash
git clone https://github.com/docker/genai-stack
docker compose --profile linux up -d
mv env.example .env
nano .env
```
In the `.env` file, make sure following lines are commented out. Set your own credentials for security
```txt
NEO4J_URI=neo4j://database:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=password
OLLAMA_BASE_URL=http://llm:11434
```
### Setting up on other platforms
For instructions on how to set up the stack on other platforms, refer to [this page](https://github.com/docker/genai-stack).
### Initial startup
The first startup may take some time because the system needs to download a large language model.
### Monitoring progress
You can monitor the download and initialization progress by viewing the logs. Run the following command to view the logs:
```bash
docker compose logs
```
Wait for specific lines in the logs indicating that the download is complete and the stack is ready. These lines typically confirm successful setup and initialization.
```text
pull-model-1 exited with code 0
database-1 | 2024-12-29 09:35:53.269+0000 INFO Started.
pdf_bot-1 | You can now view your Streamlit app in your browser.
loader-1 | You can now view your Streamlit app in your browser.
bot-1 | You can now view your Streamlit app in your browser.
```
You can now access the interface at [http://localhost:8501/](http://localhost:8501/) to ask questions. For example, you can try the sample question:
When those lines appear in the logs, the web apps are ready to use.
Since the goal is to teach AI about things it does not yet know, begin by asking it a simple question about NiFi at
[http://localhost:8501/](http://localhost:8501/).
![alt text](images/genai-leveraging-rag-1.png)
```text
Question: What is Apache Nifi?
RAG: Disabled
Hello! I'm here to help you with your question about Apache NiFi. Unfortunately, I don't know the answer to that question. I'm just an AI and my knowledge cutoff is December 2022, so I may not be familiar with the latest technologies or software. Can you please provide more context or details about Apache NiFi? Maybe there's something I can help you with related to it.
```
As shown, the AI does not know anything about this subject because it did not exist during the time of its training, also known as the information cutoff point.
Now it's time to teach the AI some new tricks. First, connect to [http://localhost:8502/](http://localhost:8502/). Instead of using the "neo4j" tag, change it to the "apache-nifi" tag, then select the **Import** button.
![alt text](images/genai-leveraging-rag-2.png)
After the import is successful, you can access Neo4j to verify the data.
After logging in to [http://localhost:7474/](http://localhost:7474/) using the credentials from the `.env` file, you can run queries on Neo4j. Using the Neo4j Cypher query language, you can check for the data stored in the database.
To count the data, run the following query:
```text
MATCH (n)
RETURN DISTINCT labels(n) AS NodeTypes, count(*) AS Count
ORDER BY Count DESC;
```
To execute this query, write in the box on the top and select the blue run button.
![alt text](images/genai-leveraging-rag-3.png)
Results will appear below. The information shown is downloaded from Stack Overflow and saved in the graph database. RAG will utilize this information to enhance its responses.
You can also run the following query to visualize the data:
```text
CALL db.schema.visualization()
```
To check the relationships in the database, run the following query:
```text
CALL db.relationshipTypes()
```
You're ready to enable the LLM to use this information. Go back to [http://localhost:8501/](http://localhost:8501/), enable the **RAG** checkbox, and ask the same question again. The LLM will provide a more detailed answer.
![alt text](images/genai-leveraging-rag-4.png)
The system delivers comprehensive, accurate information by pulling from current technical documentation.
```text
Question: What is Apache Nifi?
RAG: Enabled
Answer:
Apache NiFi is an open-source, distributed data processing system that allows you to collect, transform, and transport data across various sources and sinks. It provides a graphical user interface for designing and managing data flows, as well as a set of APIs for integrating with other systems.
According to this Stack Overflow question, the free Neo4J JDBC driver doesn't understand SQL, only Cypher, so you would need to use a non-free BI Connector to connect to Neo4j from Apache Drill.
Additionally, this Stack Overflow question suggests that Apache POI does work in a Neo4j User Defined Function. However, there may be an unknown glitch in the standalone UDF mentioned in the initial question.
To export data from Neo4j to Excel using APOC Extended procedures, you can use apoc.export.xls.query which takes a Cypher query and exports the results to an Excel file.
References:
How to connect and query Neo4j Database on Apache Drill?
Is a Neo4j UDF compatible with Apache POI?
```
Keep in mind that new questions will be added to Stack Overflow, and due to the inherent randomness in most AI models, the answers may vary and won't be identical to those in this example.
Feel free to start over with another [Stack Overflow tag](https://stackoverflow.com/tags). To drop all data in Neo4j, you can use the following command in the Neo4j Web UI:
```txt
MATCH (n)
DETACH DELETE n;
```
For optimal results, choose a tag that the LLM is not familiar with.
### When to leverage RAG for optimal results
Retrieval-Augmented Generation (RAG) is particularly effective in scenarios where standard Large Language Models (LLMs) fall short. The three key areas where RAG excels are knowledge limitations, business requirements, and cost efficiency. The following sections explore these aspects in more detail.
#### Overcoming knowledge limitations
LLMs are trained on a fixed dataset up until a certain point in time. This means they lack access to:
* Real-time information: LLMs do not continuously update their knowledge, so they may not be aware of recent events, newly released research, or emerging technologies.
* Specialized knowledge: Many niche subjects, proprietary frameworks, or industry-specific best practices may not be well-documented in the model’s training corpus.
* Accurate contextual understanding: LLMs can struggle with nuances or evolving terminologies that frequently change within dynamic fields like finance, cybersecurity, or medical research.
By incorporating RAG with a graph database such as Neo4j, AI models can access and retrieve the latest, relevant, and highly connected data before generating a response. This ensures that answers are up-to-date and grounded in factual information rather than inferred approximations.
#### Addressing business and compliance needs
Organizations in industries like healthcare, legal services, and financial analysis require their AI-driven solutions to be:
* Accurate: Businesses need AI-generated content that is factual and relevant to their specific domain.
* Compliant: Many industries must adhere to strict regulations regarding data usage and security.
* Traceable: Enterprises often require AI responses to be auditable, meaning they need to reference source material.
By using RAG, AI-generated answers can be sourced from trusted databases, ensuring higher accuracy and compliance with industry standards. This mitigates risks such as misinformation or regulatory violations.
#### Enhancing cost efficiency and performance
Training and fine-tuning large AI models can be computationally expensive and time-consuming. However, integrating RAG provides:
* Reduced fine-tuning needs: Instead of retraining an AI model every time new data emerges, RAG allows the model to fetch and incorporate updated information dynamically.
* Better performance with smaller models: With the right retrieval techniques, even compact AI models can perform well by leveraging external knowledge efficiently.
* Lower operational costs: Instead of investing in expensive infrastructure to support large-scale retraining, businesses can optimize resources by utilizing RAG’s real-time retrieval capabilities.
By following this guide, you now have the foundational knowledge to implement RAG with Neo4j, enabling your AI system to deliver more accurate, relevant, and insightful responses. The next step is experimentation—choose a dataset, configure your stack, and start enhancing your AI with the power of retrieval-augmented generation.
content/guides/genai-video-bot.md
-265
View File
@@ -1,265 +0,0 @@
---
title: GenAI video transcription and chat
linkTitle: Video transcription and chat
description: Explore a generative AI video analysis app that uses Docker, OpenAI, and Pinecone.
keywords: python, generative ai, genai, llm, whisper, pinecone, openai, whisper
summary: |
Learn how to build and deploy a generative AI video analysis and
transcription bot using Docker.
aliases:
- /guides/use-case/genai-video-bot/
params:
tags: [ai]
time: 20 minutes
---
## Overview
This guide presents a project on video transcription and analysis using a set of
technologies related to the
[GenAI Stack](https://www.docker.com/blog/introducing-a-new-genai-stack/).
The project showcases the following technologies:
- [Docker and Docker Compose](#docker-and-docker-compose)
- [OpenAI](#openai-api)
- [Whisper](#whisper)
- [Embeddings](#embeddings)
- [Chat completions](#chat-completions)
- [Pinecone](#pinecone)
- [Retrieval-Augmented Generation](#retrieval-augmented-generation)
> **Acknowledgment**
>
> This guide is a community contribution. Docker would like to thank
> [David Cardozo](https://www.davidcardozo.com/) for his contribution
> to this guide.
## Prerequisites
- You have an [OpenAI API Key](https://platform.openai.com/api-keys).
> [!NOTE]
>
> OpenAI is a third-party hosted service and [charges](https://openai.com/pricing) may apply.
- You have a [Pinecone API Key](https://app.pinecone.io/).
- You have installed the latest version of [Docker Desktop](/get-started/get-docker.md). Docker adds new features regularly and some parts of this guide may work only with the latest version of Docker Desktop.
- You have a [Git client](https://git-scm.com/downloads). The examples in this section use a command-line based Git client, but you can use any client.
## About the application
The application is a chatbot that can answer questions from a video. In
addition, it provides timestamps from the video that can help you find the sources used to answer your question.
## Get and run the application
1. Clone the sample application's repository. In a terminal, run the following
command.
```console
$ git clone https://github.com/Davidnet/docker-genai.git
```
The project contains the following directories and files:
```text
├── docker-genai/
│ ├── docker-bot/
│ ├── yt-whisper/
│ ├── .env.example
│ ├── .gitignore
│ ├── LICENSE
│ ├── README.md
│ └── docker-compose.yaml
```
2. Specify your API keys. In the `docker-genai` directory, create a text file
called `.env` and specify your API keys inside. The following is the contents of the `.env.example` file that you can refer to as an example.
```text
#-------------------------------------------------------------------------
# OpenAI
#-------------------------------------------------------------------------
OPENAI_TOKEN=your-api-key # Replace your-api-key with your personal API key
#-------------------------------------------------------------------------
# Pinecone
#-------------------------------------------------------------------------
PINECONE_TOKEN=your-api-key # Replace your-api-key with your personal API key
```
3. Build and run the application. In a terminal, change directory to your
`docker-genai` directory and run the following command.
```console
$ docker compose up --build
```
Docker Compose builds and runs the application based on the services defined
in the `docker-compose.yaml` file. When the application is running, you'll
see the logs of 2 services in the terminal.
In the logs, you'll see the services are exposed on ports `8503` and `8504`.
The two services are complimentary to each other.
The `yt-whisper` service is running on port `8503`. This service feeds the
Pinecone database with videos that you want to archive in your knowledge
database. The following section explores this service.
## Using the yt-whisper service
The yt-whisper service is a YouTube video processing service that uses the OpenAI
Whisper model to generate transcriptions of videos and stores them in a Pinecone
database. The following steps show how to use the service.
1. Open a browser and access the yt-whisper service at [http://localhost:8503](http://localhost:8503).
2. Once the application appears, in the **Youtube URL** field specify a Youtube video URL
and select **Submit**. The following example uses
[https://www.youtube.com/watch?v=yaQZFhrW0fU](https://www.youtube.com/watch?v=yaQZFhrW0fU).
![Submitting a video in the yt-whisper service](images/genai-video-bot-yt-whisper.webp)
The yt-whisper service downloads the audio of the video, uses Whisper to
transcribe it into a WebVTT (`*.vtt`) format (which you can download), then
uses the text-embedding-3-small model to create embeddings, and finally
uploads those embeddings in to the Pinecone database.
After processing the video, a video list appears in the web app that informs
you which videos have been indexed in Pinecone. It also provides a button to
download the transcript.
![A processed video in the yt-whisper service](images/genai-video-bot-yt-whisper-2.webp)
You can now access the dockerbot service on port `8504` and ask questions
about the videos.
## Using the dockerbot service
The dockerbot service is a question-answering service that leverages both the
Pinecone database and an AI model to provide responses. The following steps show
how to use the service.
> [!NOTE]
>
> You must process at least one video via the
> [yt-whisper service](#using-the-yt-whisper-service) before using
> the dockerbot service.
1. Open a browser and access the service at
[http://localhost:8504](http://localhost:8504).
2. In the **What do you want to know about your videos?** text box, ask the
Dockerbot a question about a video that was processed by the yt-whisper
service. The following example asks the question, "What is a sugar cookie?".
The answer to that question exists in the video processed in the previous
example,
[https://www.youtube.com/watch?v=yaQZFhrW0fU](https://www.youtube.com/watch?v=yaQZFhrW0fU).
![Asking a question to the Dockerbot](images/genai-video-bot-bot.webp)
In this example, the Dockerbot answers the question and
provides links to the video with timestamps, which may contain more
information about the answer.
The dockerbot service takes the question, turns it into an embedding using
the text-embedding-3-small model, queries the Pinecone database to find
similar embeddings, and then passes that context into the gpt-4-turbo-preview
to generate an answer.
3. Select the first link to see what information it provides. Based on the
previous example, select
[https://www.youtube.com/watch?v=yaQZFhrW0fU&t=553s](https://www.youtube.com/watch?v=yaQZFhrW0fU&t=553s).
In the example link, you can see that the section of video perfectly answers
the question, "What is a sugar cookie?".
## Explore the application architecture
The following image shows the application's high-level service architecture, which includes:
- yt-whisper: A local service, ran by Docker Compose, that interacts with the
remote OpenAI and Pinecone services.
- dockerbot: A local service, ran by Docker Compose, that interacts with the
remote OpenAI and Pinecone services.
- OpenAI: A remote third-party service.
- Pinecone: A remote third-party service.
![Application architecture diagram](images/genai-video-bot-architecture.webp)
## Explore the technologies used and their role
### Docker and Docker Compose
The application uses Docker to run the application in containers, providing a
consistent and isolated environment for running it. This means the application
will operate as intended within its Docker containers, regardless of the
underlying system differences. To learn more about Docker, follow the
[containerization tutorial](/get-started/tutorials/run-an-app.md).
Docker Compose is a tool for defining and running multi-container applications.
Compose makes it easy to run this application with a single command, `docker
compose up`. For more details, see the [Compose overview](/manuals/compose/_index.md).
### OpenAI API
The OpenAI API provides an LLM service that's known for its cutting-edge AI and
machine learning technologies. In this application, OpenAI's technology is used
to generate transcriptions from audio (using the Whisper model) and to create
embeddings for text data, as well as to generate responses to user queries
(using GPT and chat completions). For more details, see
[openai.com](https://openai.com/product).
### Whisper
Whisper is an automatic speech recognition system developed by OpenAI, designed
to transcribe spoken language into text. In this application, Whisper is used to
transcribe the audio from YouTube videos into text, enabling further processing
and analysis of the video content. For more details, see [Introducing Whisper](https://openai.com/research/whisper).
### Embeddings
Embeddings are numerical representations of text or other data types, which
capture their meaning in a way that can be processed by machine learning
algorithms. In this application, embeddings are used to convert video
transcriptions into a vector format that can be queried and analyzed for
relevance to user input, facilitating efficient search and response generation
in the application. For more details, see OpenAI's
[Embeddings](https://platform.openai.com/docs/guides/embeddings) documentation.
![Embedding diagram](images/genai-video-bot-embeddings.webp)
### Chat completions
Chat completion, as utilized in this application through OpenAI's API, refers to
the generation of conversational responses based on a given context or prompt.
In the application, it is used to provide intelligent, context-aware answers to
user queries by processing and integrating information from video transcriptions
and other inputs, enhancing the chatbot's interactive capabilities. For more
details, see OpenAI's
[Chat Completions API](https://platform.openai.com/docs/guides/text-generation) documentation.
### Pinecone
Pinecone is a vector database service optimized for similarity search, used for
building and deploying large-scale vector search applications. In this
application, Pinecone is employed to store and retrieve the embeddings of video
transcriptions, enabling efficient and relevant search functionality within the
application based on user queries. For more details, see
[pincone.io](https://www.pinecone.io/).
### Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) is a technique that combines information
retrieval with a language model to generate responses based on retrieved
documents or data. In RAG, the system retrieves relevant information (in this
case, via embeddings from video transcriptions) and then uses a language model
to generate responses based on this retrieved data. For more details, see
OpenAI's cookbook for
[Retrieval Augmented Generative Question Answering with Pinecone](https://cookbook.openai.com/examples/vector_databases/pinecone/gen_qa).
## Next steps
Explore how to [create a PDF bot application](/guides/genai-pdf-bot/_index.md) using
generative AI, or view more GenAI samples in the
[GenAI Stack](https://github.com/docker/genai-stack) repository.
content/guides/images/genai-leveraging-rag-1.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 15 KiB

content/guides/images/genai-leveraging-rag-2.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 192 KiB

content/guides/images/genai-leveraging-rag-3.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 30 KiB

content/guides/images/genai-leveraging-rag-4.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 16 KiB

content/guides/images/genai-video-bot-architecture.webp
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 65 KiB

content/guides/images/genai-video-bot-bot.webp
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 38 KiB

content/guides/images/genai-video-bot-embeddings.webp
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 10 KiB

content/guides/images/genai-video-bot-yt-whisper-2.webp
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 18 KiB

content/guides/images/genai-video-bot-yt-whisper.webp
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 10 KiB

content/guides/language-translation.md
-351
View File
@@ -1,351 +0,0 @@
---
title: Build a language translation app
linkTitle: Language translation
keywords: nlp, natural language processing, text summarization, python, language translation, googletrans
description: Learn how to build and run a language translation application using Python, Googletrans, and Docker.
summary: |
This guide demonstrates how to use Docker to deploy language translation
models for NLP tasks.
aliases:
- /guides/use-case/nlp/language-translation/
params:
tags: [ai]
time: 20 minutes
---
## Overview
This guide walks you through building and running a language translation
application. You'll build the application using Python with Googletrans, and
then set up the environment and run the application using Docker.
The application demonstrates a simple but practical use of the Googletrans
library for language translation, showcasing basic Python and Docker concepts.
Googletrans is a free and unlimited Python library that implements the Google
Translate API. It uses the Google Translate Ajax API to make calls to such
methods as detect and translate.
## Prerequisites
- You have installed the latest version of [Docker Desktop](/get-started/get-docker.md). Docker adds new features regularly and some parts of this guide may work only with the latest version of Docker Desktop.
- You have a [Git client](https://git-scm.com/downloads). The examples in this section use a command-line based Git client, but you can use any client.
## Get the sample application
1. Open a terminal, and clone the sample application's repository using the
following command.
```console
$ git clone https://github.com/harsh4870/Docker-NLP.git
```
2. Verify that you cloned the repository.
You should see the following files in your `Docker-NLP` directory.
```text
01_sentiment_analysis.py
02_name_entity_recognition.py
03_text_classification.py
04_text_summarization.py
05_language_translation.py
entrypoint.sh
requirements.txt
Dockerfile
README.md
```
## Explore the application code
The source code for the application is in the
`Docker-NLP/05_language_translation.py` file. Open `05_language_translation.py`
in a text or code editor to explore its contents in the following steps.
1. Import the required libraries.
```python
from googletrans import Translator
```
This line imports the `Translator` class from `googletrans`.
Googletrans is a Python library that provides an interface to Google
Translate's AJAX API.
2. Specify the main execution block.
```python
if __name__ == "__main__":
```
This Python idiom ensures that the following code block runs only if this
script is the main program. It provides flexibility, allowing the script to
function both as a standalone program and as an imported module.
3. Create an infinite loop for continuous input.
```python
while True:
input_text = input("Enter the text for translation (type 'exit' to end): ")
if input_text.lower() == 'exit':
print("Exiting...")
break
```
An infinite loop is established here to continuously prompt you for text
input, ensuring interactivity. The loop breaks when you type `exit`, allowing
you to control the application flow effectively.
4. Create an instance of Translator.
```python
translator = Translator()
```
This creates an instance of the Translator class, which
performs the translation.
5. Translate text.
```python
translated_text = translator.translate(input_text, dest='fr').text
```
Here, the `translator.translate` method is called with the user input. The
`dest='fr'` argument specifies that the destination language for translation
is French. The `.text` attribute gets the translated string. For more details
about the available language codes, see the
[Googletrans docs](https://py-googletrans.readthedocs.io/en/latest/).
6. Print the original and translated text.
```python
print(f"Original Text: {input_text}")
print(f"Translated Text: {translated_text}")
```
These two lines print the original text entered by the user and the
translated text.
7. Create `requirements.txt`. The sample application already contains the
`requirements.txt` file to specify the necessary modules that the
application imports. Open `requirements.txt` in a code or text editor to
explore its contents.
```text
...
# 05 language_translation
googletrans==4.0.0-rc1
```
Only `googletrans` is required for the language translation application.
## Explore the application environment
You'll use Docker to run the application in a container. Docker lets you
containerize the application, providing a consistent and isolated environment
for running it. This means the application will operate as intended within its
Docker container, regardless of the underlying system differences.
To run the application in a container, a Dockerfile is required. A Dockerfile is
a text document that contains all the commands you would call on the command
line to assemble an image. An image is a read-only template with instructions
for creating a Docker container.
The sample application already contains a `Dockerfile`. Open the `Dockerfile` in a code or text editor to explore its contents.
The following steps explain each part of the `Dockerfile`. For more details, see the [Dockerfile reference](/reference/dockerfile/).
1. Specify the base image.
```dockerfile
FROM python:3.8-slim
```
This command sets the foundation for the build. `python:3.8-slim` is a
lightweight version of the Python 3.8 image, optimized for size and speed.
Using this slim image reduces the overall size of your Docker image, leading
to quicker downloads and less surface area for security vulnerabilities. This
is particularly useful for a Python-based application where you might not
need the full standard Python image.
2. Set the working directory.
```dockerfile
WORKDIR /app
```
`WORKDIR` sets the current working directory within the Docker image. By
setting it to `/app`, you ensure that all subsequent commands in the
Dockerfile
(like `COPY` and `RUN`) are executed in this directory. This also helps in
organizing your Docker image, as all application-related files are contained
in a specific directory.
3. Copy the requirements file into the image.
```dockerfile
COPY requirements.txt /app
```
The `COPY` command transfers the `requirements.txt` file from
your local machine into the Docker image. This file lists all Python
dependencies required by the application. Copying it into the container
lets the next command (`RUN pip install`) install these dependencies
inside the image environment.
4. Install the Python dependencies in the image.
```dockerfile
RUN pip install --no-cache-dir -r requirements.txt
```
This line uses `pip`, Python's package installer, to install the packages
listed in `requirements.txt`. The `--no-cache-dir` option disables
the cache, which reduces the size of the Docker image by not storing the
unnecessary cache data.
5. Run additional commands.
```dockerfile
RUN python -m spacy download en_core_web_sm
```
This step is specific to NLP applications that require the spaCy library. It downloads the `en_core_web_sm` model, which is a small English language model for spaCy. While not needed for this app, it's included for compatibility with other NLP applications that might use this Dockerfile.
6. Copy the application code into the image.
```dockerfile
COPY *.py /app
COPY entrypoint.sh /app
```
These commands copy your Python scripts and the `entrypoint.sh` script into the image's `/app` directory. This is crucial because the container needs these scripts to run the application. The `entrypoint.sh` script is particularly important as it dictates how the application starts inside the container.
7. Set permissions for the `entrypoint.sh` script.
```dockerfile
RUN chmod +x /app/entrypoint.sh
```
This command modifies the file permissions of `entrypoint.sh`, making it
executable. This step is necessary to ensure that the Docker container can
run this script to start the application.
8. Set the entry point.
```dockerfile
ENTRYPOINT ["/app/entrypoint.sh"]
```
The `ENTRYPOINT` instruction configures the container to run `entrypoint.sh`
as its default executable. This means that when the container starts, it
automatically executes the script.
You can explore the `entrypoint.sh` script by opening it in a code or text
editor. As the sample contains several applications, the script lets you
specify which application to run when the container starts.
## Run the application
To run the application using Docker:
1. Build the image.
In a terminal, run the following command inside the directory of where the `Dockerfile` is located.
```console
$ docker build -t basic-nlp .
```
The following is a break down of the command:
- `docker build`: This is the primary command used to build a Docker image
from a Dockerfile and a context. The context is typically a set of files at
a specified location, often the directory containing the Dockerfile.
- `-t basic-nlp`: This is an option for tagging the image. The `-t` flag
stands for tag. It assigns a name to the image, which in this case is
`basic-nlp`. Tags are a convenient way to reference images later,
especially when pushing them to a registry or running containers.
- `.`: This is the last part of the command and specifies the build context.
The period (`.`) denotes the current directory. Docker will look for a
Dockerfile in this directory. The build context (the current directory, in
this case) is sent to the Docker daemon to enable the build. It includes
all the files and subdirectories in the specified directory.
For more details, see the [docker build CLI reference](/reference/cli/docker/buildx/build/).
Docker outputs several logs to your console as it builds the image. You'll
see it download and install the dependencies. Depending on your network
connection, this may take several minutes. Docker does have a caching
feature, so subsequent builds can be faster. The console will
return to the prompt when it's complete.
2. Run the image as a container.
In a terminal, run the following command.
```console
$ docker run -it basic-nlp 05_language_translation.py
```
The following is a break down of the command:
- `docker run`: This is the primary command used to run a new container from
a Docker image.
- `-it`: This is a combination of two options:
- `-i` or `--interactive`: This keeps the standard input (STDIN) open even
if not attached. It lets the container remain running in the
foreground and be interactive.
- `-t` or `--tty`: This allocates a pseudo-TTY, essentially simulating a
terminal, like a command prompt or a shell. It's what lets you
interact with the application inside the container.
- `basic-nlp`: This specifies the name of the Docker image to use for
creating the container. In this case, it's the image named `basic-nlp` that
you created with the `docker build` command.
- `05_language_translation.py`: This is the script you want to run inside the
Docker container. It gets passed to the `entrypoint.sh` script, which runs
it when the container starts.
For more details, see the [docker run CLI reference](/reference/cli/docker/container/run/).
> [!NOTE]
>
> For Windows users, you may get an error when running the container. Verify
> that the line endings in the `entrypoint.sh` are `LF` (`\n`) and not `CRLF` (`\r\n`),
> then rebuild the image. For more details, see [Avoid unexpected syntax errors, use Unix style line endings for files in containers](/desktop/troubleshoot-and-support/troubleshoot/topics/#unexpected-syntax-errors-use-unix-style-line-endings-for-files-in-containers).
You will see the following in your console after the container starts.
```console
Enter the text for translation (type 'exit' to end):
```
3. Test the application.
Enter some text to get the text summarization.
```console
Enter the text for translation (type 'exit' to end): Hello, how are you doing?
Original Text: Hello, how are you doing?
Translated Text: Bonjour comment allez-vous?
```
## Summary
In this guide, you learned how to build and run a language translation
application. You learned how to build the application using Python with
Googletrans, and then set up the environment and run the application using
Docker.
Related information:
- [Docker CLI reference](/reference/cli/docker/)
- [Dockerfile reference](/reference/dockerfile/)
- [Googletrans](https://github.com/ssut/py-googletrans)
- [Python documentation](https://docs.python.org/3/)
## Next steps
Explore more [natural language processing guides](./_index.md).
content/guides/named-entity-recognition.md
-356
View File
@@ -1,356 +0,0 @@
---
title: Build a named entity recognition app
linkTitle: Named entity recognition
keywords: nlp, natural language processing, named entity recognition, python, spacy, ner
description: Learn how to build and run a named entity recognition application using Python, spaCy, and Docker.
summary: |
This guide explains how to containerize named entity recognition (NER) models
using Docker.
aliases:
- /guides/use-case/nlp/named-entity-recognition/
params:
tags: [ai]
time: 20 minutes
---
## Overview
This guide walks you through building and running a named entity recognition
(NER) application. You'll build the application using Python with
spaCy, and then set up the environment and run the application using Docker.
The application processes input text to identify and print named entities, like people, organizations, or locations.
## Prerequisites
- You have installed the latest version of [Docker Desktop](/get-started/get-docker.md). Docker adds new features regularly and some parts of this guide may work only with the latest version of Docker Desktop.
- You have a [Git client](https://git-scm.com/downloads). The examples in this section use a command-line based Git client, but you can use any client.
## Get the sample application
1. Open a terminal, and clone the sample application's repository using the
following command.
```console
$ git clone https://github.com/harsh4870/Docker-NLP.git
```
2. Verify that you cloned the repository.
You should see the following files in your `Docker-NLP` directory.
```text
01_sentiment_analysis.py
02_name_entity_recognition.py
03_text_classification.py
04_text_summarization.py
05_language_translation.py
entrypoint.sh
requirements.txt
Dockerfile
README.md
```
## Explore the application code
The source code for the name recognition application is in the `Docker-NLP/02_name_entity_recognition.py` file. Open `02_name_entity_recognition.py` in a text or code editor to explore its contents in the following steps.
1. Import the required libraries.
```python
import spacy
```
This line imports the `spaCy` library. `spaCy` is a popular library in Python
used for natural language processing (NLP).
2. Load the language model.
```python
nlp = spacy.load("en_core_web_sm")
```
Here, the `spacy.load` function loads a language model. The `en_core_web_sm`
model is a small English language model. You can use this model for various
NLP tasks, including tokenization, part-of-speech tagging, and named entity
recognition.
3. Specify the main execution block.
```python
if __name__ == "__main__":
```
This Python idiom ensures that the following code block runs only if this
script is the main program. It provides flexibility, allowing the script to
function both as a standalone program and as an imported module.
4. Create an infinite loop for continuous input.
```python
while True:
```
This while loop runs indefinitely until it's explicitly broken. It lets
the user continuously enter text for entity recognition until they decide
to exit.
5. Get user input.
```python
input_text = input("Enter the text for entity recognition (type 'exit' to end): ")
```
This line prompts the user to enter text. The program will then perform entity recognition on this text.
6. Define an exit condition.
```python
if input_text.lower() == 'exit':
print("Exiting...")
break
```
If the user types something, the program converts the input to lowercase and
compares it to `exit`. If they match, the program prints **Exiting...** and
breaks out of the while loop, effectively ending the program.
7. Perform named entity recognition.
```python
doc = nlp(input_text)
for ent in doc.ents:
print(f"Entity: {ent.text}, Type: {ent.label_}")
```
- `doc = nlp(input_text)`: Here, the nlp model processes the user-input text. This creates a Doc object which contains various NLP attributes, including identified entities.
- `for ent in doc.ents:`: This loop iterates over the entities found in the text.
- `print(f"Entity: {ent.text}, Type: {ent.label_}")`: For each entity, it prints the entity text and its type (like PERSON, ORG, or GPE).
8. Create `requirements.txt`.
The sample application already contains the `requirements.txt` file to specify the necessary packages that the application imports. Open `requirements.txt` in a code or text editor to explore its contents.
```text
# 02 named_entity_recognition
spacy==3.7.2
...
```
Only the `spacy` package is required for the named recognition application.
## Explore the application environment
You'll use Docker to run the application in a container. Docker lets you
containerize the application, providing a consistent and isolated environment
for running it. This means the application will operate as intended within its
Docker container, regardless of the underlying system differences.
To run the application in a container, a Dockerfile is required. A Dockerfile is
a text document that contains all the commands you would call on the command
line to assemble an image. An image is a read-only template with instructions
for creating a Docker container.
The sample application already contains a `Dockerfile`. Open the `Dockerfile` in a code or text editor to explore its contents.
The following steps explain each part of the `Dockerfile`. For more details, see the [Dockerfile reference](/reference/dockerfile/).
1. Specify the base image.
```dockerfile
FROM python:3.8-slim
```
This command sets the foundation for the build. `python:3.8-slim` is a
lightweight version of the Python 3.8 image, optimized for size and speed.
Using this slim image reduces the overall size of your Docker image, leading
to quicker downloads and less surface area for security vulnerabilities. This
is particularly useful for a Python-based application where you might not
need the full standard Python image.
2. Set the working directory.
```dockerfile
WORKDIR /app
```
`WORKDIR` sets the current working directory within the Docker image. By
setting it to `/app`, you ensure that all subsequent commands in the
Dockerfile (like `COPY` and `RUN`) are executed in this directory. This also
helps in organizing your Docker image, as all application-related files are
contained in a specific directory.
3. Copy the requirements file into the image.
```dockerfile
COPY requirements.txt /app
```
The `COPY` command transfers the `requirements.txt` file from
your local machine into the Docker image. This file lists all Python
dependencies required by the application. Copying it into the container
lets the next command (`RUN pip install`) install these dependencies
inside the image environment.
4. Install the Python dependencies in the image.
```dockerfile
RUN pip install --no-cache-dir -r requirements.txt
```
This line uses `pip`, Python's package installer, to install the packages
listed in `requirements.txt`. The `--no-cache-dir` option disables
the cache, which reduces the size of the Docker image by not storing the
unnecessary cache data.
5. Run additional commands.
```dockerfile
RUN python -m spacy download en_core_web_sm
```
This step is specific to NLP applications that require the spaCy library. It downloads the `en_core_web_sm` model, which is a small English language model for spaCy.
6. Copy the application code into the image.
```dockerfile
COPY *.py /app
COPY entrypoint.sh /app
```
These commands copy your Python scripts and the `entrypoint.sh` script into
the image's `/app` directory. This is crucial because the container needs
these scripts to run the application. The `entrypoint.sh` script is
particularly important as it dictates how the application starts inside the
container.
7. Set permissions for the `entrypoint.sh` script.
```dockerfile
RUN chmod +x /app/entrypoint.sh
```
This command modifies the file permissions of `entrypoint.sh`, making it
executable. This step is necessary to ensure that the Docker container can
run this script to start the application.
8. Set the entry point.
```dockerfile
ENTRYPOINT ["/app/entrypoint.sh"]
```
The `ENTRYPOINT` instruction configures the container to run `entrypoint.sh`
as its default executable. This means that when the container starts, it
automatically executes the script.
You can explore the `entrypoint.sh` script by opening it in a code or text
editor. As the sample contains several applications, the script lets you
specify which application to run when the container starts.
## Run the application
To run the application using Docker:
1. Build the image.
In a terminal, run the following command inside the directory of where the `Dockerfile` is located.
```console
$ docker build -t basic-nlp .
```
The following is a break down of the command:
- `docker build`: This is the primary command used to build a Docker image
from a Dockerfile and a context. The context is typically a set of files at
a specified location, often the directory containing the Dockerfile.
- `-t basic-nlp`: This is an option for tagging the image. The `-t` flag
stands for tag. It assigns a name to the image, which in this case is
`basic-nlp`. Tags are a convenient way to reference images later,
especially when pushing them to a registry or running containers.
- `.`: This is the last part of the command and specifies the build context.
The period (`.`) denotes the current directory. Docker will look for a
Dockerfile in this directory. The build context (the current directory, in
this case) is sent to the Docker daemon to enable the build. It includes
all the files and subdirectories in the specified directory.
For more details, see the [docker build CLI reference](/reference/cli/docker/buildx/build/).
Docker outputs several logs to your console as it builds the image. You'll
see it download and install the dependencies. Depending on your network
connection, this may take several minutes. Docker does have a caching
feature, so subsequent builds can be faster. The console will
return to the prompt when it's complete.
2. Run the image as a container.
In a terminal, run the following command.
```console
$ docker run -it basic-nlp 02_name_entity_recognition.py
```
The following is a break down of the command:
- `docker run`: This is the primary command used to run a new container from
a Docker image.
- `-it`: This is a combination of two options:
- `-i` or `--interactive`: This keeps the standard input (STDIN) open even
if not attached. It lets the container remain running in the
foreground and be interactive.
- `-t` or `--tty`: This allocates a pseudo-TTY, essentially simulating a
terminal, like a command prompt or a shell. It's what lets you
interact with the application inside the container.
- `basic-nlp`: This specifies the name of the Docker image to use for
creating the container. In this case, it's the image named `basic-nlp` that
you created with the `docker build` command.
- `02_name_entity_recognition.py`: This is the script you want to run inside
the Docker container. It gets passed to the `entrypoint.sh` script, which
runs it when the container starts.
For more details, see the [docker run CLI reference](/reference/cli/docker/container/run/).
> [!NOTE]
>
> For Windows users, you may get an error when running the container. Verify
> that the line endings in the `entrypoint.sh` are `LF` (`\n`) and not `CRLF` (`\r\n`),
> then rebuild the image. For more details, see [Avoid unexpected syntax errors, use Unix style line endings for files in containers](/desktop/troubleshoot-and-support/troubleshoot/topics/#unexpected-syntax-errors-use-unix-style-line-endings-for-files-in-containers).
You will see the following in your console after the container starts.
```console
Enter the text for entity recognition (type 'exit' to end):
```
3. Test the application.
Enter some information to get the named entity recognition.
```console
Enter the text for entity recognition (type 'exit' to end): Apple Inc. is planning to open a new store in San Francisco. Tim Cook is the CEO of Apple.
Entity: Apple Inc., Type: ORG
Entity: San Francisco, Type: GPE
Entity: Tim Cook, Type: PERSON
Entity: Apple, Type: ORG
```
## Summary
This guide demonstrated how to build and run a named entity recognition
application. You learned how to build the application using Python with spaCy,
and then set up the environment and run the application using Docker.
Related information:
- [Docker CLI reference](/reference/cli/docker/)
- [Dockerfile reference](/reference/dockerfile/)
- [spaCy](https://spacy.io/)
- [Python documentation](https://docs.python.org/3/)
## Next steps
Explore more [natural language processing guides](./_index.md).
content/guides/sentiment-analysis.md
-367
View File
@@ -1,367 +0,0 @@
---
title: Build a sentiment analysis app
linkTitle: Sentiment analysis
keywords: nlp, natural language processing, sentiment analysis, python, nltk
description: Learn how to build and run a sentiment analysis application using Python, NLTK, and Docker.
summary: |
This guide demonstrates how to containerize sentiment analysis models using
Docker.
aliases:
- /guides/use-case/nlp/sentiment-analysis/
params:
tags: [ai]
time: 20 minutes
---
## Overview
In this guide, you learn how to build and run a sentiment analysis application.
You'll build the application using Python with the Natural Language Toolkit
(NLTK), and then set up the environment and run the application using Docker.
The application analyzes user input text for sentiment using NLTK's
SentimentIntensityAnalyzer and outputs whether the sentiment is positive,
negative, or neutral.
## Prerequisites
- You have installed the latest version of [Docker Desktop](/get-started/get-docker.md). Docker adds new features regularly and some parts of this guide may work only with the latest version of Docker Desktop.
- You have a [Git client](https://git-scm.com/downloads). The examples in this section use a command-line based Git client, but you can use any client.
## Get the sample application
1. Open a terminal, and clone the sample application's repository using the
following command.
```console
$ git clone https://github.com/harsh4870/Docker-NLP.git
```
2. Verify that you cloned the repository.
You should see the following files in your `Docker-NLP` directory.
```text
01_sentiment_analysis.py
02_name_entity_recognition.py
03_text_classification.py
04_text_summarization.py
05_language_translation.py
entrypoint.sh
requirements.txt
Dockerfile
README.md
```
## Explore the application code
The source code for the sentiment analysis application is in the `Docker-NLP/01_sentiment_analysis.py` file. Open `01_sentiment_analysis.py` in a text or code editor to explore its contents in the following steps.
1. Import the required libraries.
```python
import nltk
from nltk.sentiment import SentimentIntensityAnalyzer
import ssl
```
- `nltk`: This is the Natural Language Toolkit library used for working with
human language data in Python.
- `SentimentIntensityAnalyzer`: This is a specific tool from NLTK used for
determining the sentiment of a piece of text.
- `ssl`: This module provides access to Transport Layer Security (encryption)
functions used for secure web connections.
2. Handle SSL certificate verification.
```python
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
pass
else:
ssl._create_default_https_context = _create_unverified_https_context
```
This block is a workaround for certain environments where downloading data through NLTK might fail due to SSL certificate verification issues. It's telling Python to ignore SSL certificate verification for HTTPS requests.
3. Download NLTK resources.
```python
nltk.download('vader_lexicon')
nltk.download('punkt')
```
- `vader_lexicon`: This is a lexicon used by the `SentimentIntensityAnalyzer`
for sentiment analysis.
- `punkt`: This is used by NLTK for tokenizing sentences. It's necessary for
the `SentimentIntensityAnalyzer` to function correctly.
4. Create a sentiment analysis function.
```python
def perform_semantic_analysis(text):
sid = SentimentIntensityAnalyzer()
sentiment_score = sid.polarity_scores(text)
if sentiment_score['compound'] >= 0.05:
return "Positive"
elif sentiment_score['compound'] <= -0.05:
return "Negative"
else:
return "Neutral"
```
- `SentimentIntensityAnalyzer()` creates an instance of the
analyzer.
- `polarity_scores(text)` generates a sentiment score for the input text.
The function returns **Positive**, **Negative**, or **Neutral** based on the
compound score.
5. Create the main loop.
```python
if __name__ == "__main__":
while True:
input_text = input("Enter the text for semantic analysis (type 'exit' to end): ")
if input_text.lower() == 'exit':
print("Exiting...")
break
result = perform_semantic_analysis(input_text)
print(f"Sentiment: {result}")
```
This part of the script runs an infinite loop to accept user input for
analysis. If the user types `exit`, the program terminates. Otherwise, it
prints out the sentiment of the provided text.
6. Create `requirements.txt`.
The sample application already contains the
`requirements.txt` file to specify the necessary packages that the
application imports. Open `requirements.txt` in a code or text editor to
explore its contents.
```text
# 01 sentiment_analysis
nltk==3.6.5
...
```
Only the `nltk` package is required for the sentiment analysis application.
## Explore the application environment
You'll use Docker to run the application in a container. Docker lets you
containerize the application, providing a consistent and isolated environment
for running it. This means the application will operate as intended within its
Docker container, regardless of the underlying system differences.
To run the application in a container, a Dockerfile is required. A Dockerfile is
a text document that contains all the commands you would call on the command
line to assemble an image. An image is a read-only template with instructions
for creating a Docker container.
The sample application already contains a `Dockerfile`. Open the `Dockerfile` in a code or text editor to explore its contents.
The following steps explain each part of the `Dockerfile`. For more details, see the [Dockerfile reference](/reference/dockerfile/).
1. Specify the base image.
```dockerfile
FROM python:3.8-slim
```
This command sets the foundation for the build. `python:3.8-slim` is a
lightweight version of the Python 3.8 image, optimized for size and speed.
Using this slim image reduces the overall size of your Docker image, leading
to quicker downloads and less surface area for security vulnerabilities. This
is particularly useful for a Python-based application where you might not
need the full standard Python image.
2. Set the working directory.
```dockerfile
WORKDIR /app
```
`WORKDIR` sets the current working directory within the Docker image. By
setting it to `/app`, you ensure that all subsequent commands in the
Dockerfile (like `COPY` and `RUN`) are executed in this directory. This also
helps in organizing your Docker image, as all application-related files are
contained in a specific directory.
3. Copy the requirements file into the image.
```dockerfile
COPY requirements.txt /app
```
The `COPY` command transfers the `requirements.txt` file from
your local machine into the Docker image. This file lists all Python
dependencies required by the application. Copying it into the container
lets the next command (`RUN pip install`) install these dependencies
inside the image environment.
4. Install the Python dependencies in the image.
```dockerfile
RUN pip install --no-cache-dir -r requirements.txt
```
This line uses `pip`, Python's package installer, to install the packages
listed in `requirements.txt`. The `--no-cache-dir` option disables
the cache, which reduces the size of the Docker image by not storing the
unnecessary cache data.
5. Run additional commands.
```dockerfile
RUN python -m spacy download en_core_web_sm
```
This step is specific to NLP applications that require the spaCy library. It downloads the `en_core_web_sm` model, which is a small English language model for spaCy. While not needed for this app, it's included for compatibility with other NLP applications that might use this Dockerfile.
6. Copy the application code into the image.
```dockerfile
COPY *.py /app
COPY entrypoint.sh /app
```
These commands copy your Python scripts and the `entrypoint.sh` script into
the image's `/app` directory. This is crucial because the container needs
these scripts to run the application. The `entrypoint.sh` script is
particularly important as it dictates how the application starts inside the
container.
7. Set permissions for the `entrypoint.sh` script.
```dockerfile
RUN chmod +x /app/entrypoint.sh
```
This command modifies the file permissions of `entrypoint.sh`, making it
executable. This step is necessary to ensure that the Docker container can
run this script to start the application.
8. Set the entry point.
```dockerfile
ENTRYPOINT ["/app/entrypoint.sh"]
```
The `ENTRYPOINT` instruction configures the container to run `entrypoint.sh`
as its default executable. This means that when the container starts, it
automatically executes the script.
You can explore the `entrypoint.sh` script by opening it in a code or text
editor. As the sample contains several applications, the script lets you
specify which application to run when the container starts.
## Run the application
To run the application using Docker:
1. Build the image.
In a terminal, run the following command inside the directory of where the `Dockerfile` is located.
```console
$ docker build -t basic-nlp .
```
The following is a break down of the command:
- `docker build`: This is the primary command used to build a Docker image
from a Dockerfile and a context. The context is typically a set of files at
a specified location, often the directory containing the Dockerfile.
- `-t basic-nlp`: This is an option for tagging the image. The `-t` flag
stands for tag. It assigns a name to the image, which in this case is
`basic-nlp`. Tags are a convenient way to reference images later,
especially when pushing them to a registry or running containers.
- `.`: This is the last part of the command and specifies the build context.
The period (`.`) denotes the current directory. Docker will look for a
Dockerfile in this directory. The build context (the current directory, in
this case) is sent to the Docker daemon to enable the build. It includes
all the files and subdirectories in the specified directory.
Docker outputs several logs to your console as it builds the image. You'll
see it download and install the dependencies. Depending on your network
connection, this may take several minutes. Docker does have a caching
feature, so subsequent builds can be faster. The console will
return to the prompt when it's complete.
For more details, see the [docker build CLI reference](/reference/cli/docker/buildx/build/).
2. Run the image as a container.
In a terminal, run the following command.
```console
$ docker run -it basic-nlp 01_sentiment_analysis.py
```
The following is a break down of the command:
- `docker run`: This is the primary command used to run a new container from
a Docker image.
- `-it`: This is a combination of two options:
- `-i` or `--interactive`: This keeps the standard input (STDIN) open even
if not attached. It lets the container remain running in the
foreground and be interactive.
- `-t` or `--tty`: This allocates a pseudo-TTY, essentially simulating a
terminal, like a command prompt or a shell. It's what lets you
interact with the application inside the container.
- `basic-nlp`: This specifies the name of the Docker image to use for
creating the container. In this case, it's the image named `basic-nlp` that
you created with the `docker build` command.
- `01_sentiment_analysis.py`: This is the script you want to run inside the
Docker container. It gets passed to the `entrypoint.sh` script, which runs
it when the container starts.
For more details, see the [docker run CLI reference](/reference/cli/docker/container/run/).
> [!NOTE]
>
> For Windows users, you may get an error when running the container. Verify
> that the line endings in the `entrypoint.sh` are `LF` (`\n`) and not `CRLF` (`\r\n`),
> then rebuild the image. For more details, see [Avoid unexpected syntax errors, use Unix style line endings for files in containers](/desktop/troubleshoot-and-support/troubleshoot/topics/#unexpected-syntax-errors-use-unix-style-line-endings-for-files-in-containers).
You will see the following in your console after the container starts.
```console
Enter the text for semantic analysis (type 'exit' to end):
```
3. Test the application.
Enter a comment to get the sentiment analysis.
```console
Enter the text for semantic analysis (type 'exit' to end): I love containers!
Sentiment: Positive
Enter the text for semantic analysis (type 'exit' to end): I'm still learning about containers.
Sentiment: Neutral
```
## Summary
In this guide, you learned how to build and run a sentiment analysis
application. You learned how to build the application using Python with NLTK,
and then set up the environment and run the application using Docker.
Related information:
- [Docker CLI reference](/reference/cli/docker/)
- [Dockerfile reference](/reference/dockerfile/)
- [Natural Language Toolkit](https://www.nltk.org/)
- [Python documentation](https://docs.python.org/3/)
## Next steps
Explore more [natural language processing guides](./_index.md).
content/guides/tensorflowjs.md
-560
View File
@@ -1,560 +0,0 @@
---
description: Learn how to deploy pre-trained models in a TensorFlow.js web applications to perform face detection.
keywords: tensorflow.js, machine learning, ml, mediapipe, blazeface, face detection
title: Face detection with TensorFlow.js
summary: |
This guide explains how to run TensorFlow.js in Docker containers.
aliases:
- /guides/use-case/tensorflowjs/
params:
tags: [ai]
time: 20 minutes
---
This guide introduces the seamless integration of TensorFlow.js with Docker to
perform face detection. In this guide, you'll explore how to:
- Run a containerized TensorFlow.js application using Docker.
- Implement face detection in a web application with TensorFlow.js.
- Construct a Dockerfile for a TensorFlow.js web application.
- Use Docker Compose for real-time application development and updates.
- Share your Docker image on Docker Hub to facilitate deployment and extend
reach.
> **Acknowledgment**
>
> Docker would like to thank [Harsh Manvar](https://github.com/harsh4870) for
> his contribution to this guide.
## Prerequisites
- You have installed the latest version of
[Docker Desktop](/get-started/get-docker.md).
- You have a [Git client](https://git-scm.com/downloads). The examples in this
guide use a command-line based Git client, but you can use any client.
## What is TensorFlow.js?
[TensorFlow.js](https://www.tensorflow.org/js) is an open-source JavaScript
library for machine learning that enables you to train and deploy ML models in
the browser or on Node.js. It supports creating new models from scratch or using
pre-trained models, facilitating a wide range of ML applications directly in web
environments. TensorFlow.js offers efficient computation, making sophisticated
ML tasks accessible to web developers without deep ML expertise.
## Why use TensorFlow.js and Docker together?
- Environment consistency and simplified deployment: Docker packages
TensorFlow.js applications and their dependencies into containers, ensuring
consistent runs across all environments and simplifying deployment.
- Efficient development and easy scaling: Docker enhances development efficiency
with features like hot reloading and facilitates easy scaling of -
TensorFlow.js applications using orchestration tools like Kubernetes.
- Isolation and enhanced security: Docker isolates TensorFlow.js applications in
secure environments, minimizing conflicts and security vulnerabilities while
running applications with limited permissions.
## Get and run the sample application
In a terminal, clone the sample application using the following command.
```console
$ git clone https://github.com/harsh4870/TensorJS-Face-Detection
```
After cloning the application, you'll notice the application has a `Dockerfile`.
This Dockerfile lets you build and run the application locally with nothing more
than Docker.
Before you run the application as a container, you must build it into an image.
Run the following command inside the `TensorJS-Face-Detection` directory to
build an image named `face-detection-tensorjs`.
```console
$ docker build -t face-detection-tensorjs .
```
The command builds the application into an image. Depending on your network
connection, it can take several minutes to download the necessary components the
first time you run the command.
To run the image as a container, run the following command in a terminal.
```console
$ docker run -p 80:80 face-detection-tensorjs
```
The command runs the container and maps port 80 in the container to port 80 on
your system.
Once the application is running, open a web browser and access the application
at [http://localhost:80](http://localhost:80). You may need to grant access to
your webcam for the application.
In the web application, you can change the backend to use one of the following:
- WASM
- WebGL
- CPU
To stop the application, press `ctrl`+`c` in the terminal.
## About the application
The sample application performs real-time face detection using
[MediaPipe](https://developers.google.com/mediapipe/), a comprehensive framework
for building multimodal machine learning pipelines. It's specifically using the
BlazeFace model, a lightweight model for detecting faces in images.
In the context of TensorFlow.js or similar web-based machine learning
frameworks, the WASM, WebGL, and CPU backends can be used to
execute operations. Each of these backends utilizes different resources and
technologies available in modern browsers and has its strengths and limitations.
The following sections are a brief breakdown of the different backends.
### WASM
WebAssembly (WASM) is a low-level, assembly-like language with a compact binary
format that runs at near-native speed in web browsers. It allows code written in
languages like C/C++ to be compiled into a binary that can be executed in the
browser.
It's a good choice when high performance is required, and either the WebGL
backend is not supported or you want consistent performance across all devices
without relying on the GPU.
### WebGL
WebGL is a browser API that allows for GPU-accelerated usage of physics and
image processing and effects as part of the web page canvas.
WebGL is well-suited for operations that are parallelizable and can
significantly benefit from GPU acceleration, such as matrix multiplications and
convolutions commonly found in deep learning models.
### CPU
The CPU backend uses pure JavaScript execution, utilizing the device's central
processing unit (CPU). This backend is the most universally compatible and
serves as a fallback when neither WebGL nor WASM backends are available or
suitable.
## Explore the application's code
Explore the purpose of each file and their contents in the following sections.
### The index.html file
The `index.html` file serves as the frontend for the web application that
utilizes TensorFlow.js for real-time face detection from the webcam video feed.
It incorporates several technologies and libraries to facilitate machine
learning directly in the browser. It uses several TensorFlow.js libraries,
including:
- tfjs-core and tfjs-converter for core TensorFlow.js functionality and model
conversion.
- tfjs-backend-webgl, tfjs-backend-cpu, and the tf-backend-wasm script
for different computational backend options that TensorFlow.js can use for
processing. These backends allow the application to perform machine learning
tasks efficiently by leveraging the user's hardware capabilities.
- The BlazeFace library, a TensorFlow model for face detection.
It also uses the following additional libraries:
- dat.GUI for creating a graphical interface to interact with the application's
settings in real-time, such as switching between TensorFlow.js backends.
- Stats.min.js for displaying performance metrics (like FPS) to monitor the
application's efficiency during operation.
{{< accordion title="index.html" >}}
```html
<style>
body {
margin: 25px;
}
.true {
color: green;
}
.false {
color: red;
}
#main {
position: relative;
margin: 50px 0;
}
canvas {
position: absolute;
top: 0;
left: 0;
}
#description {
margin-top: 20px;
width: 600px;
}
#description-title {
font-weight: bold;
font-size: 18px;
}
</style>
<body>
<div id="main">
<video
id="video"
playsinline
style="
-webkit-transform: scaleX(-1);
transform: scaleX(-1);
width: auto;
height: auto;
"
></video>
<canvas id="output"></canvas>
<video
id="video"
playsinline
style="
-webkit-transform: scaleX(-1);
transform: scaleX(-1);
visibility: hidden;
width: auto;
height: auto;
"
></video>
</div>
</body>
<script src="https://unpkg.com/@tensorflow/tfjs-core@2.1.0/dist/tf-core.js"></script>
<script src="https://unpkg.com/@tensorflow/tfjs-converter@2.1.0/dist/tf-converter.js"></script>
<script src="https://unpkg.com/@tensorflow/tfjs-backend-webgl@2.1.0/dist/tf-backend-webgl.js"></script>
<script src="https://unpkg.com/@tensorflow/tfjs-backend-cpu@2.1.0/dist/tf-backend-cpu.js"></script>
<script src="./tf-backend-wasm.js"></script>
<script src="https://unpkg.com/@tensorflow-models/blazeface@0.0.5/dist/blazeface.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/dat-gui/0.7.6/dat.gui.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/stats.js/r16/Stats.min.js"></script>
<script src="./index.js"></script>
```
{{< /accordion >}}
### The index.js file
The `index.js` file conducts the facial detection logic. It demonstrates several
advanced concepts in web development and machine learning integration. Here's a
breakdown of some of its key components and functionalities:
- Stats.js: The script starts by creating a Stats instance to monitor and
display the frame rate (FPS) of the application in real time. This is helpful
for performance analysis, especially when testing the impact of different
TensorFlow.js backends on the application's speed.
- TensorFlow.js: The application allows users to switch between different
computation backends (wasm, webgl, and cpu) for TensorFlow.js through a
graphical interface provided by dat.GUI. Changing the backend can affect
performance and compatibility depending on the device and browser. The
addFlagLabels function dynamically checks and displays whether SIMD (Single
Instruction, Multiple Data) and multithreading are supported, which are
relevant for performance optimization in the wasm backend.
- setupCamera function: Initializes the user's webcam using the MediaDevices Web
API. It configures the video stream to not include audio and to use the
front-facing camera (facingMode: 'user'). Once the video metadata is loaded,
it resolves a promise with the video element, which is then used for face
detection.
- BlazeFace: The core of this application is the renderPrediction function,
which performs real-time face detection using the BlazeFace model, a
lightweight model for detecting faces in images. The function calls
model.estimateFaces on each animation frame to detect faces from the video
feed. For each detected face, it draws a red rectangle around the face and
blue dots for facial landmarks on a canvas overlaying the video.
{{< accordion title="index.js" >}}
```javascript
const stats = new Stats();
stats.showPanel(0);
document.body.prepend(stats.domElement);
let model, ctx, videoWidth, videoHeight, video, canvas;
const state = {
backend: "wasm",
};
const gui = new dat.GUI();
gui
.add(state, "backend", ["wasm", "webgl", "cpu"])
.onChange(async (backend) => {
await tf.setBackend(backend);
addFlagLabels();
});
async function addFlagLabels() {
if (!document.querySelector("#simd_supported")) {
const simdSupportLabel = document.createElement("div");
simdSupportLabel.id = "simd_supported";
simdSupportLabel.style = "font-weight: bold";
const simdSupported = await tf.env().getAsync("WASM_HAS_SIMD_SUPPORT");
simdSupportLabel.innerHTML = `SIMD supported: <span class=${simdSupported}>${simdSupported}<span>`;
document.querySelector("#description").appendChild(simdSupportLabel);
}
if (!document.querySelector("#threads_supported")) {
const threadSupportLabel = document.createElement("div");
threadSupportLabel.id = "threads_supported";
threadSupportLabel.style = "font-weight: bold";
const threadsSupported = await tf
.env()
.getAsync("WASM_HAS_MULTITHREAD_SUPPORT");
threadSupportLabel.innerHTML = `Threads supported: <span class=${threadsSupported}>${threadsSupported}</span>`;
document.querySelector("#description").appendChild(threadSupportLabel);
}
}
async function setupCamera() {
video = document.getElementById("video");
const stream = await navigator.mediaDevices.getUserMedia({
audio: false,
video: { facingMode: "user" },
});
video.srcObject = stream;
return new Promise((resolve) => {
video.onloadedmetadata = () => {
resolve(video);
};
});
}
const renderPrediction = async () => {
stats.begin();
const returnTensors = false;
const flipHorizontal = true;
const annotateBoxes = true;
const predictions = await model.estimateFaces(
video,
returnTensors,
flipHorizontal,
annotateBoxes,
);
if (predictions.length > 0) {
ctx.clearRect(0, 0, canvas.width, canvas.height);
for (let i = 0; i < predictions.length; i++) {
if (returnTensors) {
predictions[i].topLeft = predictions[i].topLeft.arraySync();
predictions[i].bottomRight = predictions[i].bottomRight.arraySync();
if (annotateBoxes) {
predictions[i].landmarks = predictions[i].landmarks.arraySync();
}
}
const start = predictions[i].topLeft;
const end = predictions[i].bottomRight;
const size = [end[0] - start[0], end[1] - start[1]];
ctx.fillStyle = "rgba(255, 0, 0, 0.5)";
ctx.fillRect(start[0], start[1], size[0], size[1]);
if (annotateBoxes) {
const landmarks = predictions[i].landmarks;
ctx.fillStyle = "blue";
for (let j = 0; j < landmarks.length; j++) {
const x = landmarks[j][0];
const y = landmarks[j][1];
ctx.fillRect(x, y, 5, 5);
}
}
}
}
stats.end();
requestAnimationFrame(renderPrediction);
};
const setupPage = async () => {
await tf.setBackend(state.backend);
addFlagLabels();
await setupCamera();
video.play();
videoWidth = video.videoWidth;
videoHeight = video.videoHeight;
video.width = videoWidth;
video.height = videoHeight;
canvas = document.getElementById("output");
canvas.width = videoWidth;
canvas.height = videoHeight;
ctx = canvas.getContext("2d");
ctx.fillStyle = "rgba(255, 0, 0, 0.5)";
model = await blazeface.load();
renderPrediction();
};
setupPage();
```
{{< /accordion >}}
### The tf-backend-wasm.js file
The `tf-backend-wasm.js` file is part of the
[TensorFlow.js library](https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-wasm).
It contains initialization logic for the TensorFlow.js WASM backend, some
utilities for interacting with the WASM binaries, and functions to set custom
paths for the WASM binaries.
### The tfjs-backend-wasm-simd.wasm file
The `tfjs-backend-wasm-simd.wasm` file is part of the
[TensorFlow.js library](https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-wasm).
It's a WASM binary that's used for the WebAssembly
backend, specifically optimized to utilize SIMD (Single Instruction, Multiple
Data) instructions.
## Explore the Dockerfile
In a Docker-based project, the Dockerfile serves as the foundational
asset for building your application's environment.
A Dockerfile is a text file that instructs Docker how to create an image of your
application's environment. An image contains everything you want and
need when running application, such as files, packages, and tools.
The following is the Dockerfile for this project.
```dockerfile
FROM nginx:stable-alpine3.17-slim
WORKDIR /usr/share/nginx/html
COPY . .
```
This Dockerfile defines an image that serves static content using Nginx from an
Alpine Linux base image.
## Develop with Compose
Docker Compose is a tool for defining and running multi-container Docker
applications. With Compose, you use a YAML file to configure your application's
services, networks, and volumes. In this case, the application isn't a
multi-container application, but Docker Compose has other useful features for
development, like [Compose Watch](/manuals/compose/how-tos/file-watch.md).
The sample application doesn't have a Compose file yet. To create a Compose
file, in the `TensorJS-Face-Detection` directory, create a text file named
`compose.yaml` and then add the following contents.
```yaml
services:
server:
build:
context: .
ports:
- 80:80
develop:
watch:
- action: sync
path: .
target: /usr/share/nginx/html
```
This Compose file defines a service that is built using the Dockerfile in the
same directory. It maps port 80 on the host to port 80 in the container. It also
has a `develop` subsection with the `watch` attribute that defines a list of
rules that control automatic service updates based on local file changes. For
more details about the Compose instructions, see the
[Compose file reference](/reference/compose-file/_index.md).
Save the changes to your `compose.yaml` file and then run the following command to run the application.
```console
$ docker compose watch
```
Once the application is running, open a web browser and access the application
at [http://localhost:80](http://localhost:80). You may need to grant access to
your webcam for the application.
Now you can make changes to the source code and see the changes automatically
reflected in the container without having to rebuild and rerun the container.
Open the `index.js` file and update the landmark points to be green instead of
blue on line 83.
```diff
- ctx.fillStyle = "blue";
+ ctx.fillStyle = "green";
```
Save the changes to the `index.js` file and then refresh the browser page. The
landmark points should now appear green.
To stop the application, press `ctrl`+`c` in the terminal.
## Share your image
Publishing your Docker image on Docker Hub streamlines deployment processes for
others, enabling seamless integration into diverse projects. It also promotes
the adoption of your containerized solutions, broadening their impact across the
developer ecosystem. To share your image:
1. [Sign up](https://www.docker.com/pricing?ref=Docs&refAction=DocsGuidesTensorflowjs) or sign in to [Docker Hub](https://hub.docker.com).
2. Rebuild your image to include the changes to your application. This time,
prefix the image name with your Docker ID. Docker uses the name to determine
which repository to push it to. Open a terminal and run the following
command in the `TensorJS-Face-Detection` directory. Replace `YOUR-USER-NAME`
with your Docker ID.
```console
$ docker build -t YOUR-USER-NAME/face-detection-tensorjs .
```
3. Run the following `docker push` command to push the image to Docker Hub.
Replace `YOUR-USER-NAME` with your Docker ID.
```console
$ docker push YOUR-USER-NAME/face-detection-tensorjs
```
4. Verify that you pushed the image to Docker Hub.
1. Go to [Docker Hub](https://hub.docker.com).
2. Select **My Hub** > **Repositories**.
3. View the **Last pushed** time for your repository.
Other users can now download and run your image using the `docker run` command. They need to replace `YOUR-USER-NAME` with your Docker ID.
```console
$ docker run -p 80:80 YOUR-USER-NAME/face-detection-tensorjs
```
## Summary
This guide demonstrated leveraging TensorFlow.js and Docker for face detection
in web applications. It highlighted the ease of running containerized
TensorFlow.js applications, and developing with Docker Compose for real-time
code changes. Additionally, it covered how sharing your Docker image on Docker
Hub can streamline deployment for others, enhancing the application's reach
within the developer community.
Related information:
- [TensorFlow.js website](https://www.tensorflow.org/js)
- [MediaPipe website](https://developers.google.com/mediapipe/)
- [Dockerfile reference](/reference/dockerfile/)
- [Compose file reference](/reference/compose-file/_index.md)
- [Docker CLI reference](/reference/cli/docker/)
- [Docker Blog: Accelerating Machine Learning with TensorFlow.js](https://www.docker.com/blog/accelerating-machine-learning-with-tensorflow-js-using-pretrained-models-and-docker/)
content/guides/text-classification.md
-439
View File
@@ -1,439 +0,0 @@
---
title: Build a text recognition app
linkTitle: Text classification
keywords: nlp, natural language processing, sentiment analysis, python, nltk, scikit-learn, text classification
description: Learn how to build and run a text recognition application using Python, NLTK, scikit-learn, and Docker.
summary: |
This guide details how to containerize text classification models using
Docker.
aliases:
- /guides/use-case/nlp/text-classification/
params:
tags: [ai]
time: 20 minutes
---
## Overview
In this guide, you'll learn how to create and run a text recognition
application. You'll build the application using Python with scikit-learn and the
Natural Language Toolkit (NLTK). Then you'll set up the environment and run the
application using Docker.
The application analyzes the sentiment of a user's input text using NLTK's
SentimentIntensityAnalyzer. It lets the user input text, which is then processed
to determine its sentiment, classifying it as either positive or negative. Also,
it displays the accuracy and a detailed classification report of its sentiment
analysis model based on a predefined dataset.
## Prerequisites
- You have installed the latest version of [Docker Desktop](/get-started/get-docker.md). Docker adds new features regularly and some parts of this guide may work only with the latest version of Docker Desktop.
- You have a [Git client](https://git-scm.com/downloads). The examples in this section use a command-line based Git client, but you can use any client.
## Get the sample application
1. Open a terminal, and clone the sample application's repository using the
following command.
```console
$ git clone https://github.com/harsh4870/Docker-NLP.git
```
2. Verify that you cloned the repository.
You should see the following files in your `Docker-NLP` directory.
```text
01_sentiment_analysis.py
02_name_entity_recognition.py
03_text_classification.py
04_text_summarization.py
05_language_translation.py
entrypoint.sh
requirements.txt
Dockerfile
README.md
```
## Explore the application code
The source code for the text classification application is in the `Docker-NLP/03_text_classification.py` file. Open `03_text_classification.py` in a text or code editor to explore its contents in the following steps.
1. Import the required libraries.
```python
import nltk
from nltk.sentiment import SentimentIntensityAnalyzer
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split
import ssl
```
- `nltk`: A popular Python library for natural language processing (NLP).
- `SentimentIntensityAnalyzer`: A component of `nltk` for sentiment analysis.
- `accuracy_score`, `classification_report`: Functions from scikit-learn for
evaluating the model.
- `train_test_split`: Function from scikit-learn to split datasets into
training and testing sets.
- `ssl`: Used for handling SSL certificate issues which might occur while
downloading data for `nltk`.
2. Handle SSL certificate verification.
```python
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
pass
else:
ssl._create_default_https_context = _create_unverified_https_context
```
This block is a workaround for certain environments where downloading data
through NLTK might fail due to SSL certificate verification issues. It's
telling Python to ignore SSL certificate verification for HTTPS requests.
3. Download NLTK resources.
```python
nltk.download('vader_lexicon')
```
The `vader_lexicon` is a lexicon used by the `SentimentIntensityAnalyzer` for
sentiment analysis.
4. Define text for testing and corresponding labels.
```python
texts = [...]
labels = [0, 1, 2, 0, 1, 2]
```
This section defines a small dataset of texts and their corresponding labels (0 for positive, 1 for negative, and 2 for spam).
5. Split the test data.
```python
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.2, random_state=42)
```
This part splits the dataset into training and testing sets, with 20% of data
as the test set. As this application uses a pre-trained model, it doesn't
train the model.
6. Set up sentiment analysis.
```python
sia = SentimentIntensityAnalyzer()
```
This code initializes the `SentimentIntensityAnalyzer` to analyze the
sentiment of text.
7. Generate predictions and classifications for the test data.
```python
vader_predictions = [sia.polarity_scores(text)["compound"] for text in X_test]
threshold = 0.2
vader_classifications = [0 if score > threshold else 1 for score in vader_predictions]
```
This part generates sentiment scores for each text in the test set and classifies them as positive or negative based on a threshold.
8. Evaluate the model.
```python
accuracy = accuracy_score(y_test, vader_classifications)
report_vader = classification_report(y_test, vader_classifications, zero_division='warn')
```
This part calculates the accuracy and classification report for the predictions.
9. Specify the main execution block.
```python
if __name__ == "__main__":
```
This Python idiom ensures that the following code block runs only if this
script is the main program. It provides flexibility, allowing the script to
function both as a standalone program and as an imported module.
10. Create an infinite loop for continuous input.
```python
while True:
input_text = input("Enter the text for classification (type 'exit' to end): ")
if input_text.lower() == 'exit':
print("Exiting...")
break
```
This while loop runs indefinitely until it's explicitly broken. It lets the
user continuously enter text for entity recognition until they decide to
exit.
11. Analyze the text.
```python
input_text_score = sia.polarity_scores(input_text)["compound"]
input_text_classification = 0 if input_text_score > threshold else 1
```
12. Print the VADER Classification Report and the sentiment analysis.
```python
print(f"Accuracy: {accuracy:.2f}")
print("\nVADER Classification Report:")
print(report_vader)
print(f"\nTest Text (Positive): '{input_text}'")
print(f"Predicted Sentiment: {'Positive' if input_text_classification == 0 else 'Negative'}")
```
13. Create `requirements.txt`. The sample application already contains the
`requirements.txt` file to specify the necessary packages that the
application imports. Open `requirements.txt` in a code or text editor to
explore its contents.
```text
# 01 sentiment_analysis
nltk==3.6.5
...
# 03 text_classification
scikit-learn==1.3.2
...
```
Both the `nltk` and `scikit-learn` modules are required for the text
classification application.
## Explore the application environment
You'll use Docker to run the application in a container. Docker lets you
containerize the application, providing a consistent and isolated environment
for running it. This means the application will operate as intended within its
Docker container, regardless of the underlying system differences.
To run the application in a container, a Dockerfile is required. A Dockerfile is
a text document that contains all the commands you would call on the command
line to assemble an image. An image is a read-only template with instructions
for creating a Docker container.
The sample application already contains a `Dockerfile`. Open the `Dockerfile` in a code or text editor to explore its contents.
The following steps explain each part of the `Dockerfile`. For more details, see the [Dockerfile reference](/reference/dockerfile/).
1. Specify the base image.
```dockerfile
FROM python:3.8-slim
```
This command sets the foundation for the build. `python:3.8-slim` is a
lightweight version of the Python 3.8 image, optimized for size and speed.
Using this slim image reduces the overall size of your Docker image, leading
to quicker downloads and less surface area for security vulnerabilities. This
is particularly useful for a Python-based application where you might not
need the full standard Python image.
2. Set the working directory.
```dockerfile
WORKDIR /app
```
`WORKDIR` sets the current working directory within the Docker image. By
setting it to `/app`, you ensure that all subsequent commands in the
Dockerfile (like `COPY` and `RUN`) are executed in this directory. This also
helps in organizing your Docker image, as all application-related files are
contained in a specific directory.
3. Copy the requirements file into the image.
```dockerfile
COPY requirements.txt /app
```
The `COPY` command transfers the `requirements.txt` file from
your local machine into the Docker image. This file lists all Python
dependencies required by the application. Copying it into the container
lets the next command (`RUN pip install`) install these dependencies
inside the image environment.
4. Install the Python dependencies in the image.
```dockerfile
RUN pip install --no-cache-dir -r requirements.txt
```
This line uses `pip`, Python's package installer, to install the packages
listed in `requirements.txt`. The `--no-cache-dir` option disables
the cache, which reduces the size of the Docker image by not storing the
unnecessary cache data.
5. Run additional commands.
```dockerfile
RUN python -m spacy download en_core_web_sm
```
This step is specific to NLP applications that require the spaCy library. It downloads the `en_core_web_sm` model, which is a small English language model for spaCy. While not needed for this app, it's included for compatibility with other NLP applications that might use this Dockerfile.
6. Copy the application code into the image.
```dockerfile
COPY *.py /app
COPY entrypoint.sh /app
```
These commands copy your Python scripts and the `entrypoint.sh` script into
the image's `/app` directory. This is crucial because the container needs
these scripts to run the application. The `entrypoint.sh` script is
particularly important as it dictates how the application starts inside the
container.
7. Set permissions for the `entrypoint.sh` script.
```dockerfile
RUN chmod +x /app/entrypoint.sh
```
This command modifies the file permissions of `entrypoint.sh`, making it
executable. This step is necessary to ensure that the Docker container can
run this script to start the application.
8. Set the entry point.
```dockerfile
ENTRYPOINT ["/app/entrypoint.sh"]
```
The `ENTRYPOINT` instruction configures the container to run `entrypoint.sh`
as its default executable. This means that when the container starts, it
automatically executes the script.
You can explore the `entrypoint.sh` script by opening it in a code or text
editor. As the sample contains several applications, the script lets you
specify which application to run when the container starts.
## Run the application
To run the application using Docker:
1. Build the image.
In a terminal, run the following command inside the directory of where the `Dockerfile` is located.
```console
$ docker build -t basic-nlp .
```
The following is a break down of the command:
- `docker build`: This is the primary command used to build a Docker image
from a Dockerfile and a context. The context is typically a set of files at
a specified location, often the directory containing the Dockerfile.
- `-t basic-nlp`: This is an option for tagging the image. The `-t` flag
stands for tag. It assigns a name to the image, which in this case is
`basic-nlp`. Tags are a convenient way to reference images later,
especially when pushing them to a registry or running containers.
- `.`: This is the last part of the command and specifies the build context.
The period (`.`) denotes the current directory. Docker will look for a
Dockerfile in this directory. The build context (the current directory, in
this case) is sent to the Docker daemon to enable the build. It includes
all the files and subdirectories in the specified directory.
For more details, see the [docker build CLI reference](/reference/cli/docker/buildx/build/).
Docker outputs several logs to your console as it builds the image. You'll
see it download and install the dependencies. Depending on your network
connection, this may take several minutes. Docker does have a caching
feature, so subsequent builds can be faster. The console will
return to the prompt when it's complete.
2. Run the image as a container.
In a terminal, run the following command.
```console
$ docker run -it basic-nlp 03_text_classification.py
```
The following is a break down of the command:
- `docker run`: This is the primary command used to run a new container from
a Docker image.
- `-it`: This is a combination of two options:
- `-i` or `--interactive`: This keeps the standard input (STDIN) open even
if not attached. It lets the container remain running in the
foreground and be interactive.
- `-t` or `--tty`: This allocates a pseudo-TTY, essentially simulating a
terminal, like a command prompt or a shell. It's what lets you
interact with the application inside the container.
- `basic-nlp`: This specifies the name of the Docker image to use for
creating the container. In this case, it's the image named `basic-nlp` that
you created with the `docker build` command.
- `03_text_classification.py`: This is the script you want to run inside the
Docker container. It gets passed to the `entrypoint.sh` script, which runs
it when the container starts.
For more details, see the [docker run CLI reference](/reference/cli/docker/container/run/).
> [!NOTE]
>
> For Windows users, you may get an error when running the container. Verify
> that the line endings in the `entrypoint.sh` are `LF` (`\n`) and not `CRLF` (`\r\n`),
> then rebuild the image. For more details, see [Avoid unexpected syntax errors, use Unix style line endings for files in containers](/desktop/troubleshoot-and-support/troubleshoot/topics/#unexpected-syntax-errors-use-unix-style-line-endings-for-files-in-containers).
You will see the following in your console after the container starts.
```console
Enter the text for classification (type 'exit' to end):
```
3. Test the application.
Enter some text to get the text classification.
```console
Enter the text for classification (type 'exit' to end): I love containers!
Accuracy: 1.00
VADER Classification Report:
precision recall f1-score support
0 1.00 1.00 1.00 1
1 1.00 1.00 1.00 1
accuracy 1.00 2
macro avg 1.00 1.00 1.00 2
weighted avg 1.00 1.00 1.00 2
Test Text (Positive): 'I love containers!'
Predicted Sentiment: Positive
```
## Summary
In this guide, you learned how to build and run a text classification
application. You learned how to build the application using Python with
scikit-learn and NLTK. Then you learned how to set up the environment and run
the application using Docker.
Related information:
- [Docker CLI reference](/reference/cli/docker/)
- [Dockerfile reference](/reference/dockerfile/)
- [Natural Language Toolkit](https://www.nltk.org/)
- [Python documentation](https://docs.python.org/3/)
- [scikit-learn](https://scikit-learn.org/)
## Next steps
Explore more [natural language processing guides](./_index.md).
content/guides/text-summarization.md
-361
View File
@@ -1,361 +0,0 @@
---
title: Build a text summarization app
linkTitle: Text summarization
keywords: nlp, natural language processing, text summarization, python, bert extractive summarizer
description: Learn how to build and run a text summarization application using Python, Bert Extractive Summarizer, and Docker.
summary: |
This guide shows how to containerize text summarization models with Docker.
aliases:
- /guides/use-case/nlp/text-summarization/
params:
tags: [ai]
time: 20 minutes
---
## Overview
In this guide, you'll learn how to build and run a text summarization
application. You'll build the application using Python with the Bert Extractive
Summarizer, and then set up the environment and run the application using
Docker.
The sample text summarization application uses the Bert Extractive Summarizer.
This tool utilizes the HuggingFace Pytorch transformers library to run
extractive summarizations. This works by first embedding the sentences, then
running a clustering algorithm, finding the sentences that are closest to the
cluster's centroids.
## Prerequisites
- You have installed the latest version of [Docker Desktop](/get-started/get-docker.md). Docker adds new features regularly and some parts of this guide may work only with the latest version of Docker Desktop.
- You have a [Git client](https://git-scm.com/downloads). The examples in this section use a command-line based Git client, but you can use any client.
## Get the sample application
1. Open a terminal, and clone the sample application's repository using the
following command.
```console
$ git clone https://github.com/harsh4870/Docker-NLP.git
```
2. Verify that you cloned the repository.
You should see the following files in your `Docker-NLP` directory.
```text
01_sentiment_analysis.py
02_name_entity_recognition.py
03_text_classification.py
04_text_summarization.py
05_language_translation.py
entrypoint.sh
requirements.txt
Dockerfile
README.md
```
## Explore the application code
The source code for the text summarization application is in the `Docker-NLP/04_text_summarization.py` file. Open `04_text_summarization.py` in a text or code editor to explore its contents in the following steps.
1. Import the required libraries.
```python
from summarizer import Summarizer
```
This line of code imports the `Summarizer` class from the `summarizer`
package, essential for your text summarization application. The summarizer
module implements the Bert Extractive Summarizer, leveraging the HuggingFace
Pytorch transformers library, renowned in the NLP (Natural Language
Processing) domain. This library offers access to pre-trained models like
BERT, which revolutionized language understanding tasks, including text
summarization.
The BERT model, or Bidirectional Encoder Representations from Transformers,
excels in understanding context in language, using a mechanism known as
"attention" to determine the significance of words in a sentence. For
summarization, the model embeds sentences and then uses a clustering
algorithm to identify key sentences, those closest to the centroids of these
clusters, effectively capturing the main ideas of the text.
2. Specify the main execution block.
```python
if __name__ == "__main__":
```
This Python idiom ensures that the following code block runs only if this
script is the main program. It provides flexibility, allowing the script to
function both as a standalone program and as an imported module.
3. Create an infinite loop for continuous input.
```python
while True:
input_text = input("Enter the text for summarization (type 'exit' to end): ")
if input_text.lower() == 'exit':
print("Exiting...")
break
```
An infinite loop continuously prompts you for text
input, ensuring interactivity. The loop breaks when you type `exit`, allowing
you to control the application flow effectively.
4. Create an instance of Summarizer.
```python
bert_model = Summarizer()
```
Here, you create an instance of the Summarizer class named `bert_model`. This
instance is now ready to perform the summarization task using the BERT model,
simplifying the complex processes of embedding sentences and clustering into
an accessible interface.
5. Generate and print a summary.
```python
summary = bert_model(input_text)
print(summary)
```
Your input text is processed by the bert_model instance, which then returns a
summarized version. This demonstrates the power of Python's high-level
libraries in enabling complex operations with minimal code.
6. Create `requirements.txt`. The sample application already contains the
`requirements.txt` file to specify the necessary modules that the
application imports. Open `requirements.txt` in a code or text editor to
explore its contents.
```text
...
# 04 text_summarization
bert-extractive-summarizer==0.10.1
...
torch==2.1.2
```
The `bert-extractive-summarizer` and `torch` modules are required for the
text summarization application. The summarizer module generates a summary of
the input text. This requires PyTorch because the underlying BERT model,
which is used for generating the summary, is implemented in PyTorch.
## Explore the application environment
You'll use Docker to run the application in a container. Docker lets you
containerize the application, providing a consistent and isolated environment
for running it. This means the application will operate as intended within its
Docker container, regardless of the underlying system differences.
To run the application in a container, a Dockerfile is required. A Dockerfile is
a text document that contains all the commands you would call on the command
line to assemble an image. An image is a read-only template with instructions
for creating a Docker container.
The sample application already contains a `Dockerfile`. Open the `Dockerfile` in a code or text editor to explore its contents.
The following steps explain each part of the `Dockerfile`. For more details, see the [Dockerfile reference](/reference/dockerfile/).
1. Specify the base image.
```dockerfile
FROM python:3.8-slim
```
This command sets the foundation for the build. `python:3.8-slim` is a
lightweight version of the Python 3.8 image, optimized for size and speed.
Using this slim image reduces the overall size of your Docker image, leading
to quicker downloads and less surface area for security vulnerabilities. This
is particularly useful for a Python-based application where you might not
need the full standard Python image.
2. Set the working directory.
```dockerfile
WORKDIR /app
```
`WORKDIR` sets the current working directory within the Docker image. By
setting it to `/app`, you ensure that all subsequent commands in the
Dockerfile (like `COPY` and `RUN`) are executed in this directory. This also
helps in organizing your Docker image, as all application-related files are
contained in a specific directory.
3. Copy the requirements file into the image.
```dockerfile
COPY requirements.txt /app
```
The `COPY` command transfers the `requirements.txt` file from
your local machine into the Docker image. This file lists all Python
dependencies required by the application. Copying it into the container
lets the next command (`RUN pip install`) to install these dependencies
inside the image environment.
4. Install the Python dependencies in the image.
```dockerfile
RUN pip install --no-cache-dir -r requirements.txt
```
This line uses `pip`, Python's package installer, to install the packages
listed in `requirements.txt`. The `--no-cache-dir` option disables
the cache, which reduces the size of the Docker image by not storing the
unnecessary cache data.
5. Run additional commands.
```dockerfile
RUN python -m spacy download en_core_web_sm
```
This step is specific to NLP applications that require the spaCy library. It
downloads the `en_core_web_sm` model, which is a small English language model
for spaCy. While not needed for this app, it's included for compatibility
with other NLP applications that might use this Dockerfile.
6. Copy the application code into the image.
```dockerfile
COPY *.py /app
COPY entrypoint.sh /app
```
These commands copy your Python scripts and the `entrypoint.sh` script into the image's `/app` directory. This is crucial because the container needs these scripts to run the application. The `entrypoint.sh` script is particularly important as it dictates how the application starts inside the container.
7. Set permissions for the `entrypoint.sh` script.
```dockerfile
RUN chmod +x /app/entrypoint.sh
```
This command modifies the file permissions of `entrypoint.sh`, making it
executable. This step is necessary to ensure that the Docker container can
run this script to start the application.
8. Set the entry point.
```dockerfile
ENTRYPOINT ["/app/entrypoint.sh"]
```
The `ENTRYPOINT` instruction configures the container to run `entrypoint.sh`
as its default executable. This means that when the container starts, it
automatically executes the script.
You can explore the `entrypoint.sh` script by opening it in a code or text
editor. As the sample contains several applications, the script lets you
specify which application to run when the container starts.
## Run the application
To run the application using Docker:
1. Build the image.
In a terminal, run the following command inside the directory of where the `Dockerfile` is located.
```console
$ docker build -t basic-nlp .
```
The following is a break down of the command:
- `docker build`: This is the primary command used to build a Docker image
from a Dockerfile and a context. The context is typically a set of files at
a specified location, often the directory containing the Dockerfile.
- `-t basic-nlp`: This is an option for tagging the image. The `-t` flag
stands for tag. It assigns a name to the image, which in this case is
`basic-nlp`. Tags are a convenient way to reference images later,
especially when pushing them to a registry or running containers.
- `.`: This is the last part of the command and specifies the build context.
The period (`.`) denotes the current directory. Docker will look for a
Dockerfile in this directory. The build context (the current directory, in
this case) is sent to the Docker daemon to enable the build. It includes
all the files and subdirectories in the specified directory.
For more details, see the [docker build CLI reference](/reference/cli/docker/buildx/build/).
Docker outputs several logs to your console as it builds the image. You'll
see it download and install the dependencies. Depending on your network
connection, this may take several minutes. Docker does have a caching
feature, so subsequent builds can be faster. The console will
return to the prompt when it's complete.
2. Run the image as a container.
In a terminal, run the following command.
```console
$ docker run -it basic-nlp 04_text_summarization.py
```
The following is a break down of the command:
- `docker run`: This is the primary command used to run a new container from
a Docker image.
- `-it`: This is a combination of two options:
- `-i` or `--interactive`: This keeps the standard input (STDIN) open even
if not attached. It lets the container remain running in the
foreground and be interactive.
- `-t` or `--tty`: This allocates a pseudo-TTY, essentially simulating a
terminal, like a command prompt or a shell. It's what lets you
interact with the application inside the container.
- `basic-nlp`: This specifies the name of the Docker image to use for
creating the container. In this case, it's the image named `basic-nlp` that
you created with the `docker build` command.
- `04_text_summarization.py`: This is the script you want to run inside the
Docker container. It gets passed to the `entrypoint.sh` script, which runs
it when the container starts.
For more details, see the [docker run CLI reference](/reference/cli/docker/container/run/).
> [!NOTE]
>
> For Windows users, you may get an error when running the container. Verify
> that the line endings in the `entrypoint.sh` are `LF` (`\n`) and not `CRLF` (`\r\n`),
> then rebuild the image. For more details, see [Avoid unexpected syntax errors, use Unix style line endings for files in containers](/desktop/troubleshoot-and-support/troubleshoot/topics/#unexpected-syntax-errors-use-unix-style-line-endings-for-files-in-containers).
You will see the following in your console after the container starts.
```console
Enter the text for summarization (type 'exit' to end):
```
3. Test the application.
Enter some text to get the text summarization.
```console
Enter the text for summarization (type 'exit' to end): Artificial intelligence (AI) is a branch of computer science that aims to create machines capable of intelligent behavior. These machines are designed to mimic human cognitive functions such as learning, problem-solving, and decision-making. AI technologies can be classified into two main types: narrow or weak AI, which is designed for a particular task, and general or strong AI, which possesses the ability to understand, learn, and apply knowledge across various domains. One of the most popular approaches in AI is machine learning, where algorithms are trained on large datasets to recognize patterns and make predictions.
Artificial intelligence (AI) is a branch of computer science that aims to create machines capable of intelligent behavior. These machines are designed to mimic human cognitive functions such as learning, problem-solving, and decision-making.
```
## Summary
In this guide, you learned how to build and run a text summarization
application. You learned how to build the application using Python with Bert
Extractive Summarizer, and then set up the environment and run the application
using Docker.
Related information:
- [Docker CLI reference](/reference/cli/docker/)
- [Dockerfile reference](/reference/dockerfile/)
- [Bert Extractive Summarizer](https://github.com/dmmiller612/bert-extractive-summarizer)
- [PyTorch](https://pytorch.org/)
- [Python documentation](https://docs.python.org/3/)
## Next steps
Explore more [natural language processing guides](./_index.md).