Python programming and a dataset

[ErnestTBass]

I may be repaeting myself here. I thought that I posted this thread. I have been prgramming for years in variuos langaues such as Fortran, Pascal and c.

I just do not know what I dataset is. I usuualy have a file of data thta is read in when I program, but that does not appaer to be what a dataset is.

I am not asking for it ot be exalpined here. I just want a reference thta carefully takes you through it from begging to end

Assumes thta you know nothing. Thta is all thta I want.

Any help apprecuated. Thanks in advance.

Respectfully,

ErnestTBass

[buran]

dataset is very general term. Can you explain in what context you have problem to understand what it is?

[ErnestTBass]

I was just programming a simple (for me) program in python. It was suppoed to run, but it failed. I am not sure why.
Python seems a very easy language,but the concept of a dataset it was slowing me down.

I will attach the readme file and you can see in what context it is being used. I cannot attach the file so I will have to just copy and past it.

# Evolve a neural network with a genetic algorithm

This is an example of how we can use a genetic algorithm in an attempt to find the optimal network parameters for classification tasks.

It's currently limited to only MLPs (ie. fully connected networks) and uses the Keras library to build, train and validate.

On the easy MNIST dataset, we are able to quickly find a network that reaches > 98% accuracy. On the more challenging CIFAR10 dataset, we get to 56% after 10 generations (with population 20).

For more, see this blog post: 
https://medium.com/@harvitronix/lets-evolve-a-neural-network-with-a-genetic-algorithm-code-included-8809bece164

For a more robust implementation that you can use in your projects, take a look at [Jan Liphardt's implementation, DeepEvolve](https://github.com/jliphard/DeepEvolve).

## To run

To run the brute force algorithm:

```python3 brute.py```

To run the genetic algorithm:

```python3 main.py```

You can set your network parameter choices by editing each of those files first. You can also choose whether to use the MNIST or CIFAR10 datasets. Simply set `dataset` to either `mnist` or `cifar10`.

## Credits

The genetic algorithm code is based on the code from this excellent blog post: https://lethain.com/genetic-algorithms-cool-name-damn-simple/

## Contributing

Have an optimization, idea, suggestion, bug report? Pull requests greatly appreciated!

## License

MIT

[buran]

I was just programming a simple (for me) program in python. It was suppoed to run, but it failed. I am not sure why.

If you need help with your code and why it fails - post your code in python tags, as well as full traceback in error tags.
From what you post it looks like your program is working on machine learning/image classification/computer vision.
As I suspected dataset in this case is used in very general sense.
You can google MNIST dataset or CIFAR10 dataset and CIFAR10 in Wikipedia you will see that these are 2 very large collections of labeled images. As far as I can tell they are used to train and test your model

[ErnestTBass]

Oaky, I have attached the two files. One is the zip file of the software system and the other is the output screen save of the errors that I get when I run the python programs in the way the readme file says to run them. main.py

python3 brute.py

python3 main.py

I am having major trouble attaching files. Please email to me in a private message how to do it. There seems to be no option to do it, until you at least preview the file. Then many files are rejected.

Here is the printout of the errors that occur when I run the two programs in python3.

Error:
james@james-VirtualBox:~/Desktop/neural-network-genetic-algorithm-master$ python3 brute.py
Traceback (most recent call last):
  File "brute.py", line 3, in <module>
    from network import Network
  File "/home/james/Desktop/neural-network-genetic-algorithm-master/network.py", line 4, in <module>
    from train import train_and_score
  File "/home/james/Desktop/neural-network-genetic-algorithm-master/train.py", line 8, in <module>
    from keras.datasets import mnist, cifar10
ImportError: No module named 'keras'




james@james-VirtualBox:~/Desktop/neural-network-genetic-algorithm-master$ python3 main.py
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    from optimizer import Optimizer
  File "/home/james/Desktop/neural-network-genetic-algorithm-master/optimizer.py", line 11, in <module>
    from network import Network
  File "/home/james/Desktop/neural-network-genetic-algorithm-master/network.py", line 4, in <module>
    from train import train_and_score
  File "/home/james/Desktop/neural-network-genetic-algorithm-master/train.py", line 8, in <module>
    from keras.datasets import mnist, cifar10
ImportError: No module named 'keras'
james@james-VirtualBox:~/Desktop/neural-network-genetic-algorithm-master$

Any help appreciated.

Thanks in advance.

Respectfully,

ErnestTBass

[buran]

I was just programming a simple (for me) program in python.

I would appreciate if you just tell us where this code comes from. It well help us to help you. It's obvious you didn't write this yourself - it's a downloaded probably from GitHub.

As to the error - you need to install keras first. Follow the installation instructions. If you already did then something when wrong and it's not properly installed or it is installed for some other python installation. Note that it has dependencies that you need to install first. If no other instructions - I would go and install the default backend - TensorFlow.


And again on the question of datasets - here are datasets included with keras
https://keras.io/datasets/

[ErnestTBass]

Of course I did not write the code. I am very new to Python. I do not know what keras is, but I will try to install it.

I am just trying to get the program to run and understand exactly what a dataset it. That was the original purpose of the post. The website that had this code did not say anything about keras.

I was interested in because, I used to use a lot of genetic algorithms. But we usually wrote them in Pascal.

As I said the code came from github, I will try to find the website for you.

I appreciate the trouble you take to address my question.

Respectfully,

ErnestTBass

https://blog.coast.ai/lets-evolve-a-neur...809bece164

[buran]

Of course I did not write the code. I am very new to Python.

Sorry, I misunderstood this

I was just programming a simple (for me) program in python.

I found the github link from the blog-post you shared. Here is the GitHub link https://github.com/harvitronix/neural-ne...-algorithm
Indeed, the blog post doesn't mention keras and maybe it's not the best place to start if you are not familiar with machine learning/neural networks. Neither am I.
In my previous post there is link to keras, but I guess you will need more gentle introduction to the topic

[ErnestTBass]

I installed keras and tqdm on my Ubuntu 16.04 OS.

When I run either program main.py or brute.py I got the error/warning:

I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this
TensorFlow binary was not compiled to use: AVX2

I then googled the error and got this suggestion to type:

export TF_CPP_MIN_LOG_LEVEL=2

It seems to get rid of the error.

I cannot tell you if it works, because the program is far too long in running. It takes several hours.

But what does this message mean and what does my export command do to eliminate it?

Any help appreciated.

Respectfully,

ErnestTBass

[buran]

It's not an error, just warning. It is explained here https://stackoverflow.com/a/47227886/4046632 if you didn't see it.
export TF_CPP_MIN_LOG_LEVEL=2 will set environmental variable TF_CPP_MIN_LOG_LEVEL to 2, i.e. don't show warnings. Or if you prefer - show only errors, fatal and num_severities levels. TensorFlow will check that variable when running to adjust the minimum logging level

here https://github.com/tensorflow/tensorflow.../logging.h is the definition of log levels:

Output:
INFO = 0;            // base_logging::INFO;
WARNING = 1;         // base_logging::WARNING;
ERROR = 2;           // base_logging::ERROR;
FATAL = 3;           // base_logging::FATAL;
NUM_SEVERITIES = 4;  // base_logging::NUM_SEVERITIES;