The Puzzle Challenge

Previously, I designed puzzles for GPT-3, leveraging the command line to see how well it could perform:

Despite its capabilities, I was disappointed to find that GPT-3 struggled with some puzzles and couldn't solve others at all. Now it's ChatGPT's turn to be tested. I chose a puzzle that GPT-3 had completely failed, simplifying the task by removing the command line aspect:

The goal was to map numbers to letters, forming the phrase "data science." What response did ChatGPT generate?

Unfortunately, the answer was far from accurate. Let’s explore what happens when I provide clearer instructions:

With this guidance, ChatGPT was able to arrive at the correct answer. A human would likely solve this puzzle intuitively, given the limited options. It seems that ChatGPT struggles with deduction in this scenario.

Logical Reasoning Experiment

Next, I assessed ChatGPT's ability to draw logical conclusions from a dialogue:

In the conversation, one speaker mentions their father. The only possible candidates are Douglas or Josh. A clear hint is Josh's comment, "Good job son," along with other remarks indicating family dynamics. However, despite asking ChatGPT multiple times, it consistently identified Douglas as the father:

This raises questions about the reasoning process behind its responses.

The Trick Question Challenge

I then created a trick question:

The concept is that after the first day, the 10 coins will magically transform into 20 apples, and there won't be any coins left to convert thereafter. Can ChatGPT solve this puzzle?

Out of four attempts, all responses were incorrect, with three variations in the answers. While the question could be interpreted as ambiguous, I attempted to clarify it further:

Despite multiple rephrasings, ChatGPT only answered correctly about 20% of the time, indicating a significant struggle with the task.

Summary of Findings

While ChatGPT is often praised for its capabilities, it’s crucial to recognize its shortcomings. The examples provided demonstrate that it is still a long way from matching human cognitive abilities. Interestingly, it can sometimes generate impressively sophisticated responses to complex inquiries, yet falters on simpler tasks. This suggests that ChatGPT primarily relies on interpolation, effectively addressing questions similar to those it has previously encountered. However, when faced with novel inquiries that demand extrapolation beyond its training data, it tends to struggle. The journey toward achieving true artificial general intelligence remains ongoing.