Skip to content
Snippets Groups Projects
Commit fcfcc99c authored by Colbry, Dirk's avatar Colbry, Dirk
Browse files

Merge branch 'main' into 'manar_branch1'

# Conflicts:
#   GUI_Tutorial.ipynb
parents 2dbde7fb a9ed445a
No related branches found
No related tags found
No related merge requests found
This diff is collapsed.
This diff is collapsed.
%% Cell type:markdown id:8459055e tags:
# Classification
**_NOTE_** autosklearn only will run in linux (feb 26, 2022)
Example coming from [here](https://automl.github.io/auto-sklearn/master/examples/20_basic/example_classification.html#sphx-glr-examples-20-basic-example-classification-py)
%% Cell type:code id:c69433ce tags:
``` python
# imports
from pprint import pprint
import sklearn.datasets
import sklearn.metrics
import pickle
import autosklearn.classification
```
%% Cell type:code id:2b1e1930 tags:
``` python
# split the dataset
X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = \
sklearn.model_selection.train_test_split(X, y, random_state=1)
```
%% Cell type:code id:15e5f821 tags:
``` python
# Fit the classifier
automl = autosklearn.classification.AutoSklearnClassifier(
time_left_for_this_task=120,
per_run_time_limit=30,
tmp_folder='/tmp/autosklearn_classification_example_tmp',
)
automl.fit(X_train, y_train, dataset_name='breast_cancer')
```
%% Cell type:code id:2d4e4d9f tags:
``` python
# Different Models run by autosklearn
print(automl.leaderboard())
```
%% Cell type:code id:72e580e7 tags:
``` python
# Show the different models
pprint(automl.show_models(), indent=4)
```
%% Cell type:code id:027039cd tags:
``` python
# Predict the test labels
predictions = automl.predict(X_test)
print("Accuracy score:", sklearn.metrics.accuracy_score(y_test, predictions))
```
%% Cell type:code id:acd372ea tags:
``` python
# Export the model with the highest rank
clf = automl.show_models()[7]['sklearn_classifier']
pickle.dump(clf,open('model.pickle','wb'))
```
%% Cell type:code id:a3324782 tags:
``` python
clf
```
%% Cell type:code id:021b7159 tags:
``` python
```
%% Cell type:markdown id:01d9c180 tags: %% Cell type:markdown id:01d9c180 tags:
# GAMA AutoML Tutorial # GAMA AutoML Tutorial
%% Cell type:markdown id:bf57cd23 tags: %% Cell type:markdown id:bf57cd23 tags:
**Team Boeing - Abirami Varatharajan, Brandon Hang, Deepak Ghimirey, Emily Lang, Jack Haas** **Team Boeing - Abirami Varatharajan, Brandon Hang, Deepak Ghimirey, Emily Lang, Jack Haas**
%% Cell type:markdown id:14c7d7c0 tags: %% Cell type:markdown id:14c7d7c0 tags:
<img src="https://github.com/openml-labs/gama/raw/master/images/logos/Logo-With-Grey-Name-Transparent.png" alt="W3Schools.com" width="800"> <img src="https://github.com/openml-labs/gama/raw/master/images/logos/Logo-With-Grey-Name-Transparent.png" alt="W3Schools.com" width="800">
Image source: https://github.com/openml-labs/gama/raw/master/images/logos/Logo-With-Grey-Name-Transparent.png Image source: https://github.com/openml-labs/gama/raw/master/images/logos/Logo-With-Grey-Name-Transparent.png
%% Cell type:markdown id:481fae1c tags:
## View Documentation At:
* https://openml-labs.github.io/gama/master/
* https://github.com/openml-labs/gama
* https://openml-labs.github.io/gama/master/index.html
* https://openml-labs.github.io/gama/master/user_guide/index.html#dashboard
* https://openml-labs.github.io/gama/master/api/index.html
%% Cell type:markdown id:25258b82 tags: %% Cell type:markdown id:25258b82 tags:
## Software Setup and Install ## Software Setup and Install
%% Cell type:markdown id:debdea63 tags: %% Cell type:markdown id:debdea63 tags:
Run the following command from inside an activated Conda Env to install the GAMA library with Pip: Run the following command from inside an activated Conda Env to install the GAMA library with Pip:
%% Cell type:code id:bded45a9 tags: %% Cell type:code id:bded45a9 tags:
``` python ``` python
# pip install gama # pip install gama
# *or*
# pip3 install gama
``` ```
%% Cell type:markdown id:32af5ac3 tags: %% Cell type:markdown id:32af5ac3 tags:
TODO: I think GAMA requires python=3.9? I cant get it to work. -- Dirk TODO: I think GAMA requires python=3.9? I cant get it to work. -- Dirk
%% Cell type:markdown id:192cbbad tags: %% Cell type:markdown id:192cbbad tags:
Alternatively, the source code can be downloaded from the following Github repository: Alternatively, the source code can be downloaded from the following Github repository:
https://github.com/openml-labs/gama https://github.com/openml-labs/gama
%% Cell type:markdown id:ce52d2cf tags: %% Cell type:markdown id:ce52d2cf tags:
## Accessing the Software ## Accessing the Software
%% Cell type:markdown id:ebfccd27 tags: %% Cell type:markdown id:ebfccd27 tags:
To access the GAMA library after it is installed, run the following Python import command: To access the GAMA library after it is installed, run the following Python import command:
%% Cell type:code id:00a14ad0 tags: %% Cell type:code id:00a14ad0 tags:
``` python ``` python
import gama import gama
``` ```
%% Cell type:markdown id:ad7f01c2 tags: %% Cell type:markdown id:ad7f01c2 tags:
## Overview ## Overview
%% Cell type:markdown id:bedce649 tags: %% Cell type:markdown id:bedce649 tags:
GAMA is an AutoML package that generates optimized machine learning pipelines, given specific input data and resource constraints. To simplify intensive labor work of selecting the correct algorithm, all the user needs to do is supply the data, and GAMA will automatically try to find a good machine learning pipeline by considering data preprocessing steps, machine learning algorithm, and hyperparameter configurations. GAMA can also combine multiple tuned machine learning pipelines together into an ensemble, which can help model performance. GAMA is currently restricted to classification and regression problems on tabular data. <br><br> GAMA is an AutoML package that generates optimized machine learning pipelines, given specific input data and resource constraints. To simplify intensive labor work of selecting the correct algorithm, all the user needs to do is supply the data, and GAMA will automatically try to find a good machine learning pipeline by considering data preprocessing steps, machine learning algorithm, and hyperparameter configurations. GAMA can also combine multiple tuned machine learning pipelines together into an ensemble, which can help model performance. GAMA is currently restricted to classification and regression problems on tabular data. <br><br>
For more information about GAMA, see the link below: For more information about GAMA, see the link below:
https://openml-labs.github.io/gama/master/index.html https://openml-labs.github.io/gama/master/index.html
%% Cell type:markdown id:c65233ea tags: %% Cell type:markdown id:c65233ea tags:
## Visualization ## Visualization
%% Cell type:markdown id:8a157003 tags: %% Cell type:markdown id:8a157003 tags:
In addition to its general use AutoML functionality, GAMA aims to serve AutoML researchers as well. During the optimization process, GAMA keeps an extensive log of progress made. Using this log, insight can be obtained on the behaviour of the search procedure. For example, it can produce a graph that shows pipeline fitness over time: graph of fitness over time In addition to its general use AutoML functionality, GAMA aims to serve AutoML researchers as well. During the optimization process, GAMA keeps an extensive log of progress made. Using this log, insight can be obtained on the behaviour of the search procedure. For example, it can produce a graph that shows pipeline fitness over time: graph of fitness over time
%% Cell type:markdown id:5b3d0114 tags: %% Cell type:markdown id:5b3d0114 tags:
## Code Examples ## Code Examples
%% Cell type:markdown id:6c9141bd tags: %% Cell type:markdown id:6c9141bd tags:
For detailed API documentation see the following link: For detailed API documentation see the following link:
https://openml-labs.github.io/gama/master/api/index.html https://openml-labs.github.io/gama/master/api/index.html
%% Cell type:markdown id:3427585f tags: %% Cell type:markdown id:3427585f tags:
### Classification ### Classification
Example sourced from: https://openml-labs.github.io/gama/master/user_guide/index.html#dashboard Example sourced from: https://openml-labs.github.io/gama/master/user_guide/index.html#dashboard
%% Cell type:code id:a43e62f3 tags: %% Cell type:code id:a43e62f3 tags:
``` python ``` python
from sklearn.datasets import load_breast_cancer from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss, accuracy_score from sklearn.metrics import log_loss, accuracy_score
from gama import GamaClassifier from gama import GamaClassifier
X, y = load_breast_cancer(return_X_y=True) X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split( X_train, X_test, y_train, y_test = train_test_split(
X, y, stratify=y, random_state=0 X, y, stratify=y, random_state=0
) )
automl = GamaClassifier(max_total_time=180, store="nothing", n_jobs=-1) automl = GamaClassifier(max_total_time=180, store="nothing", n_jobs=-1)
print("Starting `fit` which will take roughly 3 minutes...") print("Starting `fit` which will take roughly 3 minutes...")
automl.fit(X_train, y_train) automl.fit(X_train, y_train)
label_predictions = automl.predict(X_test) label_predictions = automl.predict(X_test)
probability_predictions = automl.predict_proba(X_test) probability_predictions = automl.predict_proba(X_test)
print("accuracy:", accuracy_score(y_test, label_predictions)) print("accuracy:", accuracy_score(y_test, label_predictions))
print("log loss:", log_loss(y_test, probability_predictions)) print("log loss:", log_loss(y_test, probability_predictions))
``` ```
%% Output %% Cell type:markdown id:1bfbb41b tags:
Starting `fit` which will take roughly 3 minutes...
--------------------------------------------------------------------------- Some issues with the classifier receiving "ValueError: population must be at least of size 3 for a pair to be selected"
IndexError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_25136/719449767.py in <module>
11 automl = GamaClassifier(max_total_time=180, store="nothing", n_jobs=-1)
12 print("Starting `fit` which will take roughly 3 minutes...")
---> 13 automl.fit(X_train, y_train)
14
15 label_predictions = automl.predict(X_test)
~\REPOS\DataTools_Tutorial_Demo\envs_gama\lib\site-packages\gama\GamaClassifier.py in fit(self, x, y, *args, **kwargs)
132 y = self._label_encoder.transform(y_)
133 self._evaluation_library.determine_sample_indices(stratify=y)
--> 134 super().fit(x, y, *args, **kwargs)
135
136 def _encode_labels(self, y):
~\REPOS\DataTools_Tutorial_Demo\envs_gama\lib\site-packages\gama\gama.py in fit(self, x, y, warm_start)
537 )
538 self._post_processing.dynamic_defaults(self)
--> 539 self.model = self._post_processing.post_process(
540 self._x,
541 self._y,
~\REPOS\DataTools_Tutorial_Demo\envs_gama\lib\site-packages\gama\postprocessing\best_fit.py in post_process(self, x, y, timeout, selection)
24 self, x: pd.DataFrame, y: pd.Series, timeout: float, selection: List[Individual]
25 ) -> object:
---> 26 self._selected_individual = selection[0]
27 return self._selected_individual.pipeline.fit(x, y)
28
IndexError: list index out of range
%% Cell type:markdown id:de9afff2 tags: %% Cell type:markdown id:de9afff2 tags:
### Regression ### Regression
Example sourced from: https://openml-labs.github.io/gama/master/user_guide/index.html#dashboard Example sourced from: https://openml-labs.github.io/gama/master/user_guide/index.html#dashboard
%% Cell type:code id:35f68b4c tags: %% Cell type:code id:35f68b4c tags:
``` python ``` python
from sklearn.datasets import load_boston from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error from sklearn.metrics import mean_squared_error
from gama import GamaRegressor from gama import GamaRegressor
X, y = load_boston(return_X_y=True) X, y = load_boston(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0) X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
automl = GamaRegressor(max_total_time=180, store="nothing", n_jobs=-1) automl = GamaRegressor(max_total_time=180, store="nothing", n_jobs=-1)
print("Starting `fit` which will take roughly 3 minutes...") print("Starting `fit` which will take roughly 3 minutes...")
automl.fit(X_train, y_train) automl.fit(X_train, y_train)
predictions = automl.predict(X_test) predictions = automl.predict(X_test)
print("MSE:", mean_squared_error(y_test, predictions)) print("MSE:", mean_squared_error(y_test, predictions))
``` ```
%% Output
Starting `fit` which will take roughly 3 minutes...
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_25136/3966760049.py in <module>
9 automl = GamaRegressor(max_total_time=180, store="nothing", n_jobs=-1)
10 print("Starting `fit` which will take roughly 3 minutes...")
---> 11 automl.fit(X_train, y_train)
12
13 predictions = automl.predict(X_test)
~\REPOS\DataTools_Tutorial_Demo\envs_gama\lib\site-packages\gama\gama.py in fit(self, x, y, warm_start)
537 )
538 self._post_processing.dynamic_defaults(self)
--> 539 self.model = self._post_processing.post_process(
540 self._x,
541 self._y,
~\REPOS\DataTools_Tutorial_Demo\envs_gama\lib\site-packages\gama\postprocessing\best_fit.py in post_process(self, x, y, timeout, selection)
24 self, x: pd.DataFrame, y: pd.Series, timeout: float, selection: List[Individual]
25 ) -> object:
---> 26 self._selected_individual = selection[0]
27 return self._selected_individual.pipeline.fit(x, y)
28
IndexError: list index out of range
%% Cell type:markdown id:a8051ac9 tags: %% Cell type:markdown id:a8051ac9 tags:
## Applications ## Applications
%% Cell type:markdown id:cc6caab0 tags: %% Cell type:markdown id:cc6caab0 tags:
This tool could be helpful when we are in the preprocessing stage and when we are comparing various models, this would be a great tool to understand which kinds of models would be best for the given data set and a create a foundation for your model. GAMA can also combine multiple tuned machine learning pipelines together into an ensemble, which should help model performance. Due to the current version of GAMA this tool currently restricted to classification and regression datasets. This tool could be helpful when we are in the preprocessing stage and when we are comparing various models, this would be a great tool to understand which kinds of models would be best for the given data set and a create a foundation for your model. GAMA can also combine multiple tuned machine learning pipelines together into an ensemble, which should help model performance. Due to the current version of GAMA this tool currently restricted to classification and regression datasets.
%% Cell type:markdown id:ffdabb07 tags:
## References
%% Cell type:markdown id:500eb430 tags:
* https://github.com/openml-labs/gama
* https://openml-labs.github.io/gama/master/index.html
* https://openml-labs.github.io/gama/master/user_guide/index.html#dashboard
* https://openml-labs.github.io/gama/master/api/index.html
%% Cell type:markdown id:e0a28b97 tags: %% Cell type:markdown id:e0a28b97 tags:
--- ---
......
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Gradio: How you can build a GUI within a Jupyter Notebook ## Gradio: How you can build a GUI within a Jupyter Notebook
#### By Team JACT #### By Team JACT
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## GUI Background ## GUI Background
A graphical user interface (GUI) is an interface through which a user interacts with electronic devices such as computers and smartphones through the use of icons, menus and other visual indicators or representations (graphics). GUIs graphically display information and related user controls, unlike text-based interfaces, where data and commands are strictly in text. GUI representations are manipulated by a pointing device such as a mouse, trackball, stylus, or by a finger on a touch screen. A graphical user interface (GUI) is an interface through which a user interacts with electronic devices such as computers and smartphones through the use of icons, menus and other visual indicators or representations (graphics). GUIs graphically display information and related user controls, unlike text-based interfaces, where data and commands are strictly in text. GUI representations are manipulated by a pointing device such as a mouse, trackball, stylus, or by a finger on a touch screen.
Source: Source:
https://www.techopedia.com/definition/5435/graphical-user-interface-gui https://www.techopedia.com/definition/5435/graphical-user-interface-gui
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Getting Started ## Getting Started
First, the gradio library must be installed on your computer. It requires Python 3.7 or later. If you have not done so already, please check your version of Python and run the following line of code: First, the gradio library must be installed on your computer. It requires Python 3.7 or later. If you have not done so already, please check your version of Python and run the following line of code:
%% Cell type:code id: tags: %% Cell type:code id:cdfdbb3e tags:
``` python ``` python
!python --version !python --version
``` ```
%% Output %% Output
Python 3.9.7 Python 3.8.2
%% Cell type:code id: tags: %% Cell type:markdown id:b8de36df tags:
On your terminal,
%% Cell type:code id:8f5edb3b tags:
``` python ``` python
#!pip install gradio #!pip install gradio
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Next, import the library as follows: Next, import the library as follows:
%% Cell type:code id: tags: %% Cell type:code id:227461d8 tags:
``` python ``` python
import gradio as gr import gradio as gr
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Gradio can be used with a wide range of media-text, pictures, video, and sound. It is most useful for demonstrating machine learning algorithms. Gradio can be used with a wide range of media-text, pictures, video, and sound. It is most useful for demonstrating machine learning algorithms.
To get a feel for how it works, run the cell below this one. An interface will automatically pop up within the Jupyter Notebook. You can type your input directing into the interface. To get a feel for how it works, run the cell below this one. An interface will automatically pop up within the Jupyter Notebook. You can type your input directing into the interface.
%% Cell type:code id: tags: %% Cell type:code id:92c83d6b tags:
``` python ``` python
def greet(name): def greet(name):
return "Hello " + name + "!!" return "Hello " + name + "!!"
iface = gr.Interface(fn=greet, inputs="text", outputs="text") iface = gr.Interface(fn=greet, inputs="text", outputs="text")
iface.launch() iface.launch()
``` ```
%% Output %% Output
Running on local URL: http://127.0.0.1:7860/ Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`. To create a public link, set `share=True` in `launch()`.
(<fastapi.applications.FastAPI at 0x7fcac199eee0>,
'http://127.0.0.1:7860/',
None)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### The Interface ### The Interface
The core interface has three parameters: The core interface has three parameters:
1. fn: The function. 1. fn: The function.
2. inputs: The input component. 2. inputs: The input component.
3. outputs: The output component. 3. outputs: The output component.
With these components, you can quickly create and launch an interface. With these components, you can quickly create and launch an interface.
%% Cell type:markdown id: tags: %% Cell type:markdown id:d4bd4273 tags:
### Adding additional Inputs:
Suppose you had a more complex function, with multiple inputs and outputs. In the example below, we define a function that takes a string, boolean, and number, and returns a string and number. Take a look how you pass a list of input and output components.
Source: https://gradio.app/quickstart/
%% Cell type:code id:dcc87cd1 tags:
``` python
def greet(name, is_morning, temperature):
salutation = "Good morning" if is_morning else "Good evening"
greeting = f"{salutation} {name}. It is {temperature} degrees today"
celsius = (temperature - 32) * 5 / 9
return greeting, round(celsius, 2)
demo = gr.Interface(
fn=greet,
inputs=["text", "checkbox", gr.Slider(0, 100)],
outputs=["text", "number"],
)
demo.launch()
```
%% Output
Running on local URL: http://127.0.0.1:7861
To create a public link, set `share=True` in `launch()`.
%% Cell type:markdown id:46e5071f tags:
### Uses of Gradio ### Uses of Gradio
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Gradio can load in data, similar to pandas frames, by using the command `gradio.inputs.Dataframe(data_name)`. It can only take in strings, numbers, bools, and dates as data types. Gradio does not contain a library of datasets, so data must be input by the user. It can also work with time series, images, audio, video, and generic file uploads. Gradio can load in data, similar to pandas frames, by using the command `gradio.inputs.Dataframe(data_name)`. It can only take in strings, numbers, bools, and dates as data types. Gradio does not contain a library of datasets, so data must be input by the user. It can also work with time series, images, audio, video, and generic file uploads.
Applications: Applications:
- Machine learning interface - Machine learning interface
- Image classification. - Image classification.
- Text generation interface (e.g. ChatBot). - Text generation interface (e.g. ChatBot).
- Audio and video editing - Audio and video editing
- Reverse audio files. - Reverse audio files.
- Flip video files. - Flip video files.
- Using machine learning, gradio can detect the main note in an inputted audio file. - Using machine learning, gradio can detect the main note in an inputted audio file.
- File Outputs: - File Outputs:
- Zip files directly within Python. - Zip files directly within Python.
- Output your data in various file formats including JSON, HTML, PNG, etc. - Output your data in various file formats including JSON, HTML, PNG, etc.
- Using a function called `.Carousel()`, Gradio can output a set of components that can be easily scrolled through. - Using a function called `.Carousel()`, Gradio can output a set of components that can be easily scrolled through.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
An example of a question could be, what is the conclusion of the sentence..... An example of a question could be, what is the conclusion of the sentence: Today will be a good day.
Today will be a good day.
For that you would be using a text generation model called GPT-2 and the example given user's Text Interface. A user could use Gardio to create an input field, then use the information from that field to generate a response. For example, the input could be sent to a text generation model such as GPT-2 to generate a response.
For a visual from data, you could create a cat/dog image classification model and Gradio demo to upload new images for class prediction. The model will be a Keras convolutional neural network (CNN) that would be trained on images of cats and dogs as features and their class names as labels For a visual from data, you could create a cat/dog image classification model and Gradio demo to upload new images for class prediction. The model will be a Keras convolutional neural network (CNN) that would be trained on images of cats and dogs as features and their class names as labels
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
How this data or tool could be used in some of the team projects (maybe not your own) How this data or tool could be used in some of the team projects (maybe not your own)
- For projects with time series modeling - For projects with time series modeling
* Hope Village * Hope Village
* Argonne * Argonne
(Include example of implemntation here)
- For image analysis - For image analysis
* Neogen * Neogen
* Ford * Ford
(Include example of implemntation here)
- Machine learning group - Machine learning group
* Kelloggs * Kelloggs
* AFRL * AFRL
(Include example of implemntation here)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Sources:<br> Sources:<br>
[Analytics Vidhya](https://www.analyticsvidhya.com/blog/2021/04/create-interface-for-your-machine-learning-models-using-gradio-python-library/) [Analytics Vidhya](https://www.analyticsvidhya.com/blog/2021/04/create-interface-for-your-machine-learning-models-using-gradio-python-library/)
[Gradio Documentation](https://gradio.app/getting_started/) [Gradio Documentation](https://gradio.app/getting_started/)
......
...@@ -33,10 +33,12 @@ ...@@ -33,10 +33,12 @@
" 1. This can be found at https://console.developers.google.com/projectselector2/apis/credentials?pli=1&supportedpurview=project \n", " 1. This can be found at https://console.developers.google.com/projectselector2/apis/credentials?pli=1&supportedpurview=project \n",
" (you must use your personal email not your MSU email)\n", " (you must use your personal email not your MSU email)\n",
"2. Inside that project enable the Google sheets API and Google Drive for your new project\n", "2. Inside that project enable the Google sheets API and Google Drive for your new project\n",
" 1. This is found by using the search bar to search for Google API and Google Drive\n", " 1. Use the search bar to find and click on the Google Sheets API.\n",
" 2. Enable the API by selecting the blue button.\n",
" 3. Repeat the first two steps for the Google Drive API.\n",
"3. Create a Google service account through the Google developer portal\n", "3. Create a Google service account through the Google developer portal\n",
"4. Go to “APIs & Services > Credentials” and choose “Create credentials > Service account key”.\n", "4. Go to “APIs & Services > Credentials” and choose “Create credentials > Service account key”.\n",
"5. Fill out the form (making sure to add editor privledges to the service account)\n", "5. Once you open the "Create service account" screen, fill out the name and ID fields and click "Create and Continue" to move to the next section. After filling out "1. Service account details" under "2. Grant this service account access to project" navigate to the "Role" drop down and choose "Basic" -> 'Editor." After you select the role, hit "Continue." You can skip the next section, "Grant users access to this service account" and click done. \n",
"6. Click “Create” and “Done”.\n", "6. Click “Create” and “Done”.\n",
"7. Press “Manage service accounts” above Service Accounts.\n", "7. Press “Manage service accounts” above Service Accounts.\n",
"8. Press on ⋮ near recently created service account and select “Manage keys” and then click on “ADD KEY > Create new key”.\n", "8. Press on ⋮ near recently created service account and select “Manage keys” and then click on “ADD KEY > Create new key”.\n",
...@@ -229,7 +231,7 @@ ...@@ -229,7 +231,7 @@
"provenance": [] "provenance": []
}, },
"kernelspec": { "kernelspec": {
"display_name": "Python 3", "display_name": "Python 3 (ipykernel)",
"language": "python", "language": "python",
"name": "python3" "name": "python3"
}, },
...@@ -243,7 +245,7 @@ ...@@ -243,7 +245,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.7.1" "version": "3.8.12"
} }
}, },
"nbformat": 4, "nbformat": 4,
......
{ {
"nbformat": 4, <<<<<<< HEAD
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Ford_Video_Analysis.ipynb",
"provenance": [],
"collapsed_sections": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [ "cells": [
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {
"id": "OqSmFS3lZj9Y"
},
"source": [ "source": [
"# <center> Video Image Data </center>\n", "# <center> Video Image Data </center>\n",
"#### CMSE 495 Ford Group\n", "#### CMSE 495 Ford Group\n",
"\n", "\n",
"This tutorial teaches the user how to input a video file, such a mp4 and convert each frame of the video into a jpeg image using python, primarily in a Jupyter notebook." "This tutorial teaches the user how to input a video file, such a mp4 and convert each frame of the video into a jpeg image using python, primarily in a Jupyter notebook."
], ]
"metadata": {
"id": "OqSmFS3lZj9Y"
}
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"source": [
"[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pathakis/DataTools_Tutorial_Demo/blob/main/Video-Image-Data-Tutorial/Ford_Video_Analysis.ipynb)"
],
"metadata": { "metadata": {
"id": "uey1neRTkRwd" "id": "uey1neRTkRwd"
} },
"source": [
"[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pathakis/DataTools_Tutorial_Demo/blob/main/Video-Image-Data-Tutorial/Ford_Video_Analysis.ipynb)"
]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {
"id": "GQIJxk_kdjT1"
},
"source": [ "source": [
"<b> Environment Setup (Makefile):</b>\n", "<b> Environment Setup (Makefile):</b>\n",
"- Use the command 'make innit' automatically set up the environment for you.\n", "- Use the command 'make innit' automatically set up the environment for you.\n",
...@@ -61,33 +49,35 @@ ...@@ -61,33 +49,35 @@
"<b> Usage Instructions:</b>\n", "<b> Usage Instructions:</b>\n",
"\n", "\n",
"- The example call shows the format in which this func may be used.\n" "- The example call shows the format in which this func may be used.\n"
], ]
"metadata": {
"id": "GQIJxk_kdjT1"
}
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"source": [
"This process uses 2 packages called [os](https://docs.python.org/3/library/os.html) and [cv2](https://pypi.org/project/opencv-python/). Os provides miscellaneous operating system interfaces such as opening and reading the files."
],
"metadata": { "metadata": {
"id": "f0aUW4PLdobE" "id": "f0aUW4PLdobE"
} },
"source": [
"This process uses 2 packages called [os](https://docs.python.org/3/library/os.html) and [cv2](https://pypi.org/project/opencv-python/). Os provides miscellaneous operating system interfaces such as opening and reading the files."
]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "execution_count": null,
"# !pip install opencv-python"
],
"metadata": { "metadata": {
"id": "JGkN_k3BgXV8" "id": "JGkN_k3BgXV8"
}, },
"execution_count": null, "outputs": [],
"outputs": [] "source": [
"# !pip install opencv-python"
]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PCbpVR-HZzmt"
},
"outputs": [],
"source": [ "source": [
"import cv2\n", "import cv2\n",
"import os\n", "import os\n",
...@@ -128,28 +118,29 @@ ...@@ -128,28 +118,29 @@
" \n", " \n",
" #releasing the threads\n", " #releasing the threads\n",
" vidcap.release()\n" " vidcap.release()\n"
], ]
"metadata": {
"id": "PCbpVR-HZzmt"
},
"execution_count": null,
"outputs": []
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {
"id": "hBfLcvMhh6v7"
},
"source": [ "source": [
"DEMO FOR THE **avi_to_frames** \n", "DEMO FOR THE **avi_to_frames** \n",
"\n", "\n",
"1. To download a sample avi file that you want to work with, use the following code `urllib.request.urlretrieve('https://www.engr.colostate.edu/me/facil/dynamics/files/drop.avi', 'testing.mp4')` \n", "1. To download a sample avi file that you want to work with, use the following code `urllib.request.urlretrieve('https://file-examples.com/wp-content/uploads/2018/04/file_example_AVI_480_750kB.avi', 'testing.mp4')` \n",
"\n", "\n",
"2. After the video has been downloaded `avi_frames(./testing.mp4, path_to_where_you_want_the_frames, False)` this will create a folder with frames from the video." "2. After the video has been downloaded `avi_frames(./testing.mp4, path_to_where_you_want_the_frames, False)` this will create a folder with frames from the video."
], ]
"metadata": {
"id": "hBfLcvMhh6v7"
}
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-N-hJD11jJjo"
},
"outputs": [],
"source": [ "source": [
"# Making a Video From Frames\n", "# Making a Video From Frames\n",
"def frames_to_video(directory_path, fps, width, height):\n", "def frames_to_video(directory_path, fps, width, height):\n",
...@@ -163,15 +154,13 @@ ...@@ -163,15 +154,13 @@
"\n", "\n",
" cv2.destroyAllWindows()\n", " cv2.destroyAllWindows()\n",
" video.release()" " video.release()"
], ]
"metadata": {
"id": "-N-hJD11jJjo"
},
"execution_count": null,
"outputs": []
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {
"id": "sIhvc4DplLkE"
},
"source": [ "source": [
"DEMO FOR THE **frames_to_video** \n", "DEMO FOR THE **frames_to_video** \n",
"\n", "\n",
...@@ -180,46 +169,317 @@ ...@@ -180,46 +169,317 @@
"2. Run the following command (**make the necessary changes in the function call**)`frames_to_video(where_the_frames_are, fps, width, height)`\n", "2. Run the following command (**make the necessary changes in the function call**)`frames_to_video(where_the_frames_are, fps, width, height)`\n",
"\n", "\n",
"3. The video will show up in the current directory." "3. The video will show up in the current directory."
], ]
"metadata": {
"id": "sIhvc4DplLkE"
}
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"source": [
"**The code below will put the image arrays into a list.** This snippet of code utilizes glob but packages like os can also be used."
],
"metadata": { "metadata": {
"id": "bnfzxPNJeZVS" "id": "bnfzxPNJeZVS"
} },
"source": [
"**The code below will put the image arrays into a list.** This snippet of code utilizes glob but packages like os can also be used."
]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mLbqPW7SeJd-"
},
"outputs": [],
"source": [ "source": [
"path = glob.glob(\"./*.jpg\")\n", "path = glob.glob(\"./*.jpg\")\n",
"images = []\n", "images = []\n",
"for img in path:\n", "for img in path:\n",
" n = cv2.imread(img)\n", " n = cv2.imread(img)\n",
" images.append(n)" " images.append(n)"
], ]
"metadata": {
"id": "mLbqPW7SeJd-"
},
"execution_count": null,
"outputs": []
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {
"id": "GIq-_h4wdxWM"
},
"source": [ "source": [
"<b> References:</b>\n", "<b> References:</b>\n",
"- [Managing Environments](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)\n", "- [Managing Environments](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)\n",
"- [Open CV in python](https://pypi.org/project/opencv-python/)\n", "- [Open CV in python](https://pypi.org/project/opencv-python/)\n",
"- [Colab Button](https://www.youtube.com/watch?v=RoGZIbwzG5w)" "- [Colab Button](https://www.youtube.com/watch?v=RoGZIbwzG5w)"
], ]
"metadata": { }
"id": "GIq-_h4wdxWM" ],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "Ford_Video_Analysis.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.9.15"
},
"vscode": {
"interpreter": {
"hash": "4f3567101b31d35f97cf2856951ebbba1e09a3f852422478e736adda2bb3beee"
} }
} }
] },
} "nbformat": 4,
\ No newline at end of file "nbformat_minor": 0
=======
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "OqSmFS3lZj9Y"
},
"source": [
"# <center> Video Image Data </center>\n",
"#### CMSE 495 Ford Group\n",
"\n",
"This tutorial teaches the user how to input a video file, such a mp4 and convert each frame of the video into a jpeg image using python, primarily in a Jupyter notebook."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uey1neRTkRwd"
},
"source": [
"[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pathakis/DataTools_Tutorial_Demo/blob/main/Video-Image-Data-Tutorial/Ford_Video_Analysis.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GQIJxk_kdjT1"
},
"source": [
"<b> Environment Setup (Makefile):</b>\n",
"- Use the command 'make innit' automatically set up the environment for you.\n",
"\n",
"<b> Environment Setup (Manual):</b>\n",
"- Set up new environment using pip/conda (Conda Recommended). Use command \n",
"\n",
" <code> conda create -n envs python=3.10 </code>\n",
"\n",
"- Activate your new environment. Use command \n",
"\n",
" <code> conda activate envs</code>\n",
"\n",
"- Install the requisite packages.Use command \n",
"\n",
" <code> pip install opencv-python</code> or,\n",
"\n",
" <code> conda install -c conda-forge opencv</code>\n",
"\n",
"<b> Usage Instructions:</b>\n",
"\n",
"- The example call shows the format in which this func may be used.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f0aUW4PLdobE"
},
"source": [
"This process uses 2 packages called [os](https://docs.python.org/3/library/os.html) and [cv2](https://pypi.org/project/opencv-python/). Os provides miscellaneous operating system interfaces such as opening and reading the files."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JGkN_k3BgXV8"
},
"outputs": [],
"source": [
"# !pip install opencv-python"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "PCbpVR-HZzmt"
},
"outputs": [],
"source": [
"import cv2\n",
"import os\n",
"import glob\n",
"import urllib.request\n",
"\n",
"\n",
"def video_to_frames(file_path, directory_path, greyscale = False):\n",
"\n",
" '''This function will change a video file to a frames'''\n",
" \n",
" #opening the video\n",
" vidcap = cv2.VideoCapture(file_path) \n",
" \n",
" dirname = directory_path\n",
" os.makedirs(dirname, exist_ok=True)\n",
" \n",
" #capturing a frame as well as a boolean value representing whether an image was properly opened\n",
" success,image = vidcap.read()\n",
" \n",
" count = 0\n",
" \n",
" while success:\n",
" \n",
" #this is specifically for foam_segmented.avi\n",
" if greyscale:\n",
" image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n",
" \n",
" #writing the image to a the directory path that was specified, \n",
" #if the path specified does not exist then it will be created\n",
" #this finctionality was added so that the images could be stored in a separate folder\n",
" #example of output file names: 1.jpg, 2.jpg, 3.jpg, and so on\n",
" cv2.imwrite(os.path.join(dirname, str(count)+\".jpg\"), image)\n",
" success,image = vidcap.read()\n",
" count += 1\n",
" #All the frames will be added in order\n",
" cv2.waitKey(1) \n",
" \n",
" #releasing the threads\n",
" vidcap.release()\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hBfLcvMhh6v7"
},
"source": [
"DEMO FOR THE **avi_to_frames** \n",
"\n",
"1. To download a sample avi file that you want to work with, use the following code `urllib.request.urlretrieve('https://www.engr.colostate.edu/me/facil/dynamics/files/drop.avi', 'testing.mp4')` \n",
"\n",
"2. After the video has been downloaded `avi_frames(./testing.mp4, path_to_where_you_want_the_frames, False)` this will create a folder with frames from the video."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "-N-hJD11jJjo"
},
"outputs": [],
"source": [
"# Making a Video From Frames\n",
"def frames_to_video(directory_path, fps, width, height):\n",
" fourcc = cv2.VideoWriter_fourcc(*'mp4v')\n",
" video = cv2.VideoWriter('video.avi', fourcc, fps, (width, height))\n",
" num_frames = len([name for name in os.listdir(directory_path) if os.path.isfile(name)])\n",
"\n",
" for j in range(num_frames):\n",
" img = cv2.imread(str(j) + '.jpg')\n",
" video.write(img)\n",
"\n",
" cv2.destroyAllWindows()\n",
" video.release()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sIhvc4DplLkE"
},
"source": [
"DEMO FOR THE **frames_to_video** \n",
"\n",
"1. We will be working with the frames that we created using `avi_to_frames`. If you have not created those frames feel free to look at the steps above.\n",
"\n",
"2. Run the following command (**make the necessary changes in the function call**)`frames_to_video(where_the_frames_are, fps, width, height)`\n",
"\n",
"3. The video will show up in the current directory."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bnfzxPNJeZVS"
},
"source": [
"**The code below will put the image arrays into a list.** This snippet of code utilizes glob but packages like os can also be used."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "mLbqPW7SeJd-"
},
"outputs": [],
"source": [
"path = glob.glob(\"./*.jpg\")\n",
"images = []\n",
"for img in path:\n",
" n = cv2.imread(img)\n",
" images.append(n)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GIq-_h4wdxWM"
},
"source": [
"<b> References:</b>\n",
"- [Managing Environments](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)\n",
"- [Open CV in python](https://pypi.org/project/opencv-python/)\n",
"- [Colab Button](https://www.youtube.com/watch?v=RoGZIbwzG5w)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Comments:**\n",
"\n",
"The videos is not working, it only downloaded a unavailable video"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "Ford_Video_Analysis.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 1
>>>>>>> 5c328fa (this is a testing comment)
}
This diff is collapsed.
source diff could not be displayed: it is too large. Options to address this: view the blob.
#### Requirements for YouTube scraper
pandas
matplotlib
pytchat
time
\ No newline at end of file
This diff is collapsed.
This diff is collapsed.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment