How to fix list index out of range

[longmen]

I'm pretty sure that is wrong. Going down from node 2 or node 3 should not take you to node 1.

No it is not but it is the closest to the solution I am looking for :/ Also, I do not think 1 here as a Node. It is a value of 1. Since None will not let it go up or right and 0 is not acceptable. So 1 works.

[deanhystad]

I am stepping up from pretty sure to I know it is wrong. When you make a node the numbers in the list are the node numbers for the neighbors of the node. For example, Node 8 has the following neighbors:

up = 12, right = out of range, down = 4 left = 7

You correctly create node 8 like this:

Node_8 = Node(8, [12, self.wall, 4, 7])

Where self.wall is None.

You correctly create nodes 6, 7, 8, 10, 11, 12, 13 and 14 using this same pattern. Specifying the node number for each neighbor and using None when there is not a neighbor in that direction.

You incorrectly create nodes 1, 2, 3, 4, 5, 9, 15 and 16. In each of these nodes moving in a direction that takes you off the board (out of the environment) magically teleports you back to node 1

The error is not immediately obvious. Node 1 is tucked away in a corner and passing through node 1 usually isn't the shortest path to the goal, but look at what happens if I move the start point up to node 12.

Output:
1 2 5 3 p
1 goal
2 goal
3 wall-square
4 right
5 forbid
6 down
7 left
8 down
9 left
10 up
11 up
12 up
13 up
14 up
15 up
16 up

Node 4 moves to the right to use the teleport pad to node 1. Nodes 10 through 16 also take advantage of the teleporter.

Your program has a lot of errors. The errors in setting up the node neighbors are the worst. These can make your program return incorrect results.

You have other errors that don't affect results. This code works fine now, but would blow up if your environment was larger than 4 x 4:

position = 0
while position < LEVEL:
    if current_episode.next[position] is not None:
        current_episode.move.insert(position,
                                    Direction(position)), current_episode.qValues.insert(
            position, False)
    position += 1

The loop should execute 4 times because there are 4 Directions, not because the environment is 4 nodes wide or 4 nodes high. The code should be this:

for direction in Direction:
    if current_episode.next[direction.value] is not None:
        current_episode.move[direction.value] = direction
        current_episode.qValues[direction.value] = False

Or even better, make direction an IntEnum instead of an Enum. Now you can use Direction like it is an integer.

class Direction(enum.IntEnum):
    up = 0
    right = 1
    down = 2
    left = 3
...
for direction in Direction:
    if current_episode.next[direction] is not None:
        current_episode.move[direction] = direction
        current_episode.qValues[direction] = False

And you have bugs that will never cause a problem, but they do not leave a good impression with those reading your code.

You define these constants but they are never used

HEIGHT = 4
WEIGHT = 4

You pass "print_best_actions" and "index" arguments to Q_learning but they are not used.

Q_learning computes a "total_episode_reward" that is never used for anything.

[longmen]

I am stepping up from pretty sure to I know it is wrong. When you make a node the numbers in the list are the node numbers for the neighbors of the node. For example, Node 8 has the following neighbors:

up = 12, right = out of range, down = 4 left = 7

You correctly create node 8 like this:

Node_8 = Node(8, [12, self.wall, 4, 7])

Where self.wall is None.

You correctly create nodes 6, 7, 8, 10, 11, 12, 13 and 14 using this same pattern. Specifying the node number for each neighbor and using None when there is not a neighbor in that direction.

You incorrectly create nodes 1, 2, 3, 4, 5, 9, 15 and 16. In each of these nodes moving in a direction that takes you off the board (out of the environment) magically teleports you back to node 1

The error is not immediately obvious. Node 1 is tucked away in a corner and passing through node 1 usually isn't the shortest path to the goal, but look at what happens if I move the start point up to node 12.

Output:
1 2 5 3 p
1 goal
2 goal
3 wall-square
4 right
5 forbid
6 down
7 left
8 down
9 left
10 up
11 up
12 up
13 up
14 up
15 up
16 up

Node 4 moves to the right to use the teleport pad to node 1. Nodes 10 through 16 also take advantage of the teleporter.

Your program has a lot of errors. The errors in setting up the node neighbors are the worst. These can make your program return incorrect results.

You have other errors that don't affect results. This code works fine now, but would blow up if your environment was larger than 4 x 4:

position = 0
while position < LEVEL:
    if current_episode.next[position] is not None:
        current_episode.move.insert(position,
                                    Direction(position)), current_episode.qValues.insert(
            position, False)
    position += 1

The loop should execute 4 times because there are 4 Directions, not because the environment is 4 nodes wide or 4 nodes high. The code should be this:

for direction in Direction:
    if current_episode.next[direction.value] is not None:
        current_episode.move[direction.value] = direction
        current_episode.qValues[direction.value] = False

Or even better, make direction an IntEnum instead of an Enum. Now you can use Direction like it is an integer.

class Direction(enum.IntEnum):
    up = 0
    right = 1
    down = 2
    left = 3
...
for direction in Direction:
    if current_episode.next[direction] is not None:
        current_episode.move[direction] = direction
        current_episode.qValues[direction] = False

And you have bugs that will never cause a problem, but they do not leave a good impression with those reading your code.

You define these constants but they are never used

HEIGHT = 4
WEIGHT = 4

You pass "print_best_actions" and "index" arguments to Q_learning but they are not used.

Q_learning computes a "total_episode_reward" that is never used for anything.

I am appreciated your help. I will carefully review it.

[longmen]

I am stepping up from pretty sure to I know it is wrong. When you make a node the numbers in the list are the node numbers for the neighbors of the node. For example, Node 8 has the following neighbors:

up = 12, right = out of range, down = 4 left = 7

You correctly create node 8 like this:

Node_8 = Node(8, [12, self.wall, 4, 7])

Where self.wall is None.

You correctly create nodes 6, 7, 8, 10, 11, 12, 13 and 14 using this same pattern. Specifying the node number for each neighbor and using None when there is not a neighbor in that direction.

You incorrectly create nodes 1, 2, 3, 4, 5, 9, 15 and 16. In each of these nodes moving in a direction that takes you off the board (out of the environment) magically teleports you back to node 1

The error is not immediately obvious. Node 1 is tucked away in a corner and passing through node 1 usually isn't the shortest path to the goal, but look at what happens if I move the start point up to node 12.

Output:
1 2 5 3 p
1 goal
2 goal
3 wall-square
4 right
5 forbid
6 down
7 left
8 down
9 left
10 up
11 up
12 up
13 up
14 up
15 up
16 up

Node 4 moves to the right to use the teleport pad to node 1. Nodes 10 through 16 also take advantage of the teleporter.

Your program has a lot of errors. The errors in setting up the node neighbors are the worst. These can make your program return incorrect results.

You have other errors that don't affect results. This code works fine now, but would blow up if your environment was larger than 4 x 4:

position = 0
while position < LEVEL:
    if current_episode.next[position] is not None:
        current_episode.move.insert(position,
                                    Direction(position)), current_episode.qValues.insert(
            position, False)
    position += 1

The loop should execute 4 times because there are 4 Directions, not because the environment is 4 nodes wide or 4 nodes high. The code should be this:

for direction in Direction:
    if current_episode.next[direction.value] is not None:
        current_episode.move[direction.value] = direction
        current_episode.qValues[direction.value] = False

Or even better, make direction an IntEnum instead of an Enum. Now you can use Direction like it is an integer.

class Direction(enum.IntEnum):
    up = 0
    right = 1
    down = 2
    left = 3
...
for direction in Direction:
    if current_episode.next[direction] is not None:
        current_episode.move[direction] = direction
        current_episode.qValues[direction] = False

And you have bugs that will never cause a problem, but they do not leave a good impression with those reading your code.

You define these constants but they are never used

HEIGHT = 4
WEIGHT = 4

You pass "print_best_actions" and "index" arguments to Q_learning but they are not used.

Q_learning computes a "total_episode_reward" that is never used for anything.

Hi so I received feedback that most parts of my implementation are correct. The only problem that I am having is to implement the position changing (the function to choose next move) of the q-values. For example, case "10 8 9 6 q 2" #2's up is a wall square and its down is out of the down edge, if the agent's next move hits these positions, it should go back to its last position, but this is not the case in my implementation, and in fact, they were treated only as "normal squares". After choosing a new direction of the four by the position changing function, the agent will potentially arrive at a new position, if it's not the wall square or out of any edge, the new position remains, otherwise, it should be put back to its last position.
I wonder if you have any insights on how to debug this? Thanks

[deanhystad]

There are lots of articles online. This one is solving a problem identical to yours.

https://www.learndatasci.com/tutorials/r...penai-gym/

[longmen]

There are lots of articles online. This one is solving a problem identical to yours.

https://www.learndatasci.com/tutorials/r...penai-gym/

I still cannot figure out how to set the agent to stay at the same position if hit a wall or boundary in my move function :(

[deanhystad]

Don't allow selecting a direction that leads directly to a wall.