{"id":441,"date":"2026-02-22T22:47:25","date_gmt":"2026-02-22T22:47:25","guid":{"rendered":"https:\/\/ffritze.de\/?p=441"},"modified":"2026-06-02T10:31:39","modified_gmt":"2026-06-02T10:31:39","slug":"q-learning-entwicklung-mit-scratch","status":"publish","type":"post","link":"https:\/\/ffritze.de\/en\/q-learning-entwicklung-mit-scratch\/","title":{"rendered":"Q-Learning \u2013 Development with Scratch"},"content":{"rendered":"<p class=\"wp-block-paragraph\">In this article we outline the development of the QField with Scratch. We start by drawing the playing field and implementing the random movement of Scratch. In the next step, we introduce the Q table, which serves as the basis for reinforcing learning. Finally, we visualize Scratch's decisions through drawn arrows that represent his path on the playing field.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Drawing the playing field<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To draw the playing field for the QField, you can either use a pre-made card or create the playing field yourself. To do this, you need three objects: the cat Scratch, the object \"cake\" for the target field and another object, ideally the \"Pencil\", which is used for drawing and remains hidden.<\/p>\n\n\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-baaca9a6 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:40%\">\n<div class=\"responsive-iframe\">\n<iframe src=\"https:\/\/scratch.mit.edu\/projects\/1282288152\/embed\" allowtransparency=\"true\" frameborder=\"0\"\nscrolling=\"no\"\nallowfullscreen>\n<\/iframe>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:50%\">\n<p class=\"wp-block-paragraph\">First, you implement the x and y lists that are responsible for field numbering. These lists make it easier to query whether the \"cake\" has been reached and support the movement of Scratch.<\/p>\n<\/div>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">The variable \"step\" is used for the width of the squares and later also for the movement of Scratch.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the next step, we will implement the random movement of Scratch and introduce the Q table.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/scratch.mit.edu\/projects\/1282288152\" target=\"_blank\" rel=\"noreferrer noopener\">\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-1212b163 wp-block-group-is-layout-flex\" style=\"margin-top:0;margin-bottom:0;padding-top:0;padding-right:0;padding-bottom:0;padding-left:0\"> <figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"48\" height=\"48\" srcset=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png 48w, https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com-12x12.png 12w\" sizes=\"auto, (max-width: 48px) 100vw, 48px\" class=\"wp-image-638\" src=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png\" alt=\"\"> <\/figure> <div class=\"wp-block-group is-vertical is-content-justification-center is-layout-flex wp-container-core-group-is-layout-81b84b2d wp-block-group-is-layout-flex\"><p class=\"has-small-font-size wp-block-paragraph\">Show the<\/p> <p class=\"has-medium-font-size wp-block-paragraph\">code<\/p> <\/div> <\/div>\n<\/a><\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Scratch runs randomly<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Now it's time to program Scratch. Here we use our own blocks for the movements \"up\", \"down\", \"left\" and \"right\". Since we have numbered the fields, when running to the right, the field number is usually increased by one, decreased to the left by one, down by 5 and to the top by 5.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-baaca9a6 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:50%\">\n<p class=\"wp-block-paragraph\">Special care is required at the edges, as exceptions must be programmed here. This works most clearly with modulo calculations, but you can also link all field positions in the margin with \"or\" to check the conditions.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:70%\">\n<div class=\"wp-block-group has-custom-hellblau-transparent-background-color has-background has-global-padding is-layout-constrained wp-container-core-group-is-layout-404298a2 wp-block-group-is-layout-constrained\" style=\"border-width:1px;border-top-left-radius:25px;border-top-right-radius:25px;border-bottom-left-radius:25px;border-bottom-right-radius:25px;margin-top:0;margin-bottom:0;padding-top:var(--wp--preset--spacing--20);padding-right:var(--wp--preset--spacing--20);padding-bottom:var(--wp--preset--spacing--20);padding-left:var(--wp--preset--spacing--20);box-shadow:var(--wp--preset--shadow--natural)\">\n<p class=\"wp-block-paragraph\">The Modulo calculation is helpful to check if Scratch is located at the edges of the playing field. With the Modulo operation, you can determine whether a field number is divisible by 5, for example, indicating that Scratch is in the last row of a 5\u00d75 field. This technique makes it possible to efficiently control Scratch's movements and ensure that it does not go beyond the boundaries of the playing field.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">In addition, a direction of movement must be randomly selected until the \"cake\" is reached. This random selection ensures that Scratch navigates the playing field and learns how to reach his goal.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-baaca9a6 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:65%\">\n<div class=\"responsive-iframe\">\n<iframe src=\"https:\/\/scratch.mit.edu\/projects\/1282292654\/embed\" allowtransparency=\"true\" frameborder=\"0\" scrolling=\"no\" allowfullscreen><\/iframe>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-vertically-aligned-bottom is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:35%\">\n<div class=\"wp-block-buttons is-content-justification-center is-layout-flex wp-container-core-buttons-is-layout-3189c7ba wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/scratch.mit.edu\/projects\/1282292654\" target=\"_blank\" rel=\"noreferrer noopener\">\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-1212b163 wp-block-group-is-layout-flex\" style=\"margin-top:0;margin-bottom:0;padding-top:0;padding-right:0;padding-bottom:0;padding-left:0\"> <figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"48\" height=\"48\" srcset=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png 48w, https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com-12x12.png 12w\" sizes=\"auto, (max-width: 48px) 100vw, 48px\" class=\"wp-image-638\" src=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png\" alt=\"\"> <\/figure> <div class=\"wp-block-group is-vertical is-content-justification-center is-layout-flex wp-container-core-group-is-layout-81b84b2d wp-block-group-is-layout-flex\"><p class=\"has-small-font-size wp-block-paragraph\">Show the<\/p> <p class=\"has-medium-font-size wp-block-paragraph\">code<\/p> <\/div> <\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Scratch learns a way<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This section is about how Scratch learns the way to his goal through reinforcing learning. To develop a better understanding of this concept, we recommend that you read&nbsp;<a href=\"https:\/\/ffritze.de\/en\/q-learning-unplugged\/\" target=\"_blank\" rel=\"noreferrer noopener\">Q-Learning unplugged.<\/a> There, the basics of reinforcement learning are clearly explained and provide a valuable basis for understanding how Scratch can adapt its decisions to achieve its goal more efficiently.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-baaca9a6 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:50%\">\n<p class=\"wp-block-paragraph\">Since reading the arrows is too complex for the computer, we introduce the Q table. However, since the Scratch programming environment only allows one-dimensional lists, a separate list is created for each direction. In the initialization of the object \"Pencil\", all fields must be presented with 0. Then a 1 is entered in the position of the \"cake\" to mark the target.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:70%\">\n<div class=\"wp-block-group has-custom-hellblau-transparent-background-color has-background has-global-padding is-layout-constrained wp-container-core-group-is-layout-38c38467 wp-block-group-is-layout-constrained\" style=\"border-width:1px;border-top-left-radius:25px;border-top-right-radius:25px;border-bottom-left-radius:25px;border-bottom-right-radius:25px;padding-top:var(--wp--preset--spacing--20);padding-right:var(--wp--preset--spacing--20);padding-bottom:var(--wp--preset--spacing--20);padding-left:var(--wp--preset--spacing--20);box-shadow:var(--wp--preset--shadow--natural)\">\n<p class=\"wp-block-paragraph\">The Q table is a central element in amplifying learning, as it stores the values for the various actions that Scratch can perform in any state. Each cell in the table represents the value of a specific movement (up, down, left, right) for a specific field. A value of 1 indicates that Scratch knows the way, while a value of 0 means that he has not yet explored this path. By continuously updating the Q-table, Scratch learns which movements lead him to his goal more efficiently<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-baaca9a6 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:60%\">\n<div class=\"responsive-iframe\">\n<iframe src=\"https:\/\/scratch.mit.edu\/projects\/1282299189\/embed\" allowtransparency=\"true\" frameborder=\"0\" scrolling=\"no\" allowfullscreen><\/iframe>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:50%\">\n<p class=\"wp-block-paragraph\">The Scratch object gets the most extensive changes. At each step, the tables must be read out. If Scratch reaches a field with a 1 in one of the direction lists, it means that he is either at the finish line or knows the way. <\/p>\n<\/div>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">In this case, a 1 must be written in the number of the previous field in the corresponding direction list, the direction that has just been taken. Therefore, it is important to run a corresponding variable that tracks the previous position. A separate block is created for the list work, which we call \"QBot\".<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/scratch.mit.edu\/projects\/1282299189\" target=\"_blank\" rel=\"noreferrer noopener\">\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-1212b163 wp-block-group-is-layout-flex\" style=\"margin-top:0;margin-bottom:0;padding-top:0;padding-right:0;padding-bottom:0;padding-left:0\"> <figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"48\" height=\"48\" srcset=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png 48w, https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com-12x12.png 12w\" sizes=\"auto, (max-width: 48px) 100vw, 48px\" class=\"wp-image-638\" src=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png\" alt=\"\"> <\/figure> <div class=\"wp-block-group is-vertical is-content-justification-center is-layout-flex wp-container-core-group-is-layout-81b84b2d wp-block-group-is-layout-flex\"><p class=\"has-small-font-size wp-block-paragraph\">Show the<\/p> <p class=\"has-medium-font-size wp-block-paragraph\">code<\/p> <\/div> <\/div>\n<\/a><\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Show the way<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To visually represent the path that Scratch has learned, we draw arrows on the map. These arrows represent the directions of movement (up, down, left, right) that Scratch can perform in each field. For each field for which a 1 is entered in a direction list, a corresponding arrow is drawn on the map, which indicates the direction of travel. This visualization not only helps to understand the learned path, but also makes it clear how Scratch makes its decisions to reach its goal efficiently.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-baaca9a6 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:65%\">\n<div class=\"responsive-iframe\">\n<iframe src=\"https:\/\/scratch.mit.edu\/projects\/1282301588\/embed\" allowtransparency=\"true\" frameborder=\"0\" scrolling=\"no\" allowfullscreen><\/iframe>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-vertically-aligned-bottom is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:35%\">\n<div class=\"wp-block-buttons is-content-justification-center is-layout-flex wp-container-core-buttons-is-layout-3189c7ba wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/scratch.mit.edu\/projects\/1282301588\" target=\"_blank\" rel=\"noreferrer noopener\">\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-1212b163 wp-block-group-is-layout-flex\" style=\"margin-top:0;margin-bottom:0;padding-top:0;padding-right:0;padding-bottom:0;padding-left:0\"> <figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"48\" height=\"48\" srcset=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png 48w, https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com-12x12.png 12w\" sizes=\"auto, (max-width: 48px) 100vw, 48px\" class=\"wp-image-638\" src=\"https:\/\/ffritze.de\/wp-content\/uploads\/2026\/03\/scratch-svgrepo-com.png\" alt=\"\"> <\/figure> <div class=\"wp-block-group is-vertical is-content-justification-center is-layout-flex wp-container-core-group-is-layout-81b84b2d wp-block-group-is-layout-flex\"><p class=\"has-small-font-size wp-block-paragraph\">Show the<\/p> <p class=\"has-medium-font-size wp-block-paragraph\">code<\/p> <\/div> <\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-group has-custom-hellblau-transparent-background-color has-background has-global-padding is-layout-constrained wp-container-core-group-is-layout-c9a371c5 wp-block-group-is-layout-constrained\" style=\"border-width:1px;border-top-left-radius:25px;border-top-right-radius:25px;border-bottom-left-radius:25px;border-bottom-right-radius:25px;padding-top:var(--wp--preset--spacing--30);padding-right:var(--wp--preset--spacing--30);padding-bottom:var(--wp--preset--spacing--30);padding-left:var(--wp--preset--spacing--30);box-shadow:var(--wp--preset--shadow--natural)\">\n<h2 class=\"wp-block-heading\">Further possibilities<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">There are numerous further opportunities to expand and improve the QField. An interesting option is the implementation of negative feedback (negative reinforcement) when Scratch runs against a wall. In this case, the value in the Q-table for this movement could be reduced, so that Scratch learns to avoid this direction.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In addition, various map shapes and obstacles can be introduced to increase the complexity of the game and make learning more challenging. Graphical improvements could also be integrated to enrich the visual experience.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An exciting example of the application of reinforcement learning is the scenario of a Mars rover looking for the drop-off location of supply packages. Here, the rover could navigate through obstacles and learn which paths are best suited to reach its goals.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>In diesem Beitrag skizzieren wir die Entwicklung des QFelds mit Scratch. Wir beginnen mit dem Zeichnen des Spielfelds und der Implementierung der zuf\u00e4lligen Bewegung von Scratch. Im n\u00e4chsten Schritt f\u00fchren wir die Q-Tabelle ein, die als Grundlage f\u00fcr das verst\u00e4rkende Lernen dient. Abschlie\u00dfend visualisieren wir die Entscheidungen von Scratch durch gezeichnete Pfeile, die seinen Weg [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":442,"comment_status":"open","ping_status":"open","sticky":false,"template":"seite-thoughts-beitrag","format":"standard","meta":{"footnotes":""},"categories":[19,7,9],"tags":[36,37],"class_list":["post-441","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-algorithms","category-programs","category-scratch","tag-maschinelles-lernen","tag-reinforcement-learning"],"_links":{"self":[{"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/posts\/441","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/comments?post=441"}],"version-history":[{"count":0,"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/posts\/441\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/media\/442"}],"wp:attachment":[{"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/media?parent=441"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/categories?post=441"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ffritze.de\/en\/wp-json\/wp\/v2\/tags?post=441"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}