SURF 2020: Test and Evaluation for Autonomy - Revision history

Murray at 21:32, 15 December 2019

2019-12-15T21:32:51Z

← Older revision		Revision as of 21:32, 15 December 2019
Line 1:		Line 1:
			'''[[SURF 2020\|2020 SURF]]: project description'''
			* Mentor: Richard M. Murray
			* Co-mentor: Apurva Badithela

	Autonomous systems are an emerging technology with potential for growth and impact in safety-critical applications such as self-driving cars, space missions, distributed power grid. In these applications, a rigorous, proof-based framework for design, test and evaluation of autonomy is necessary.		Autonomous systems are an emerging technology with potential for growth and impact in safety-critical applications such as self-driving cars, space missions, distributed power grid. In these applications, a rigorous, proof-based framework for design, test and evaluation of autonomy is necessary.

Abadithe at 02:02, 11 December 2019

2019-12-11T02:02:59Z

← Older revision		Revision as of 02:02, 11 December 2019
Line 34:		Line 34:
	5. Wongpiromsarn, Tichakorn, et al. "TuLiP: a software toolbox for receding horizon temporal logic planning." Proceedings of the 14th international conference on Hybrid systems: computation and control. ACM, 2011. [https://user.eng.umd.edu/~mumu/files/wtoxm_HSCC2011.pdf Link]		5. Wongpiromsarn, Tichakorn, et al. "TuLiP: a software toolbox for receding horizon temporal logic planning." Proceedings of the 14th international conference on Hybrid systems: computation and control. ACM, 2011. [https://user.eng.umd.edu/~mumu/files/wtoxm_HSCC2011.pdf Link]

	6. Breach, ~~A Toolbox~~ for ~~Verification~~ and ~~Parameter Synthesis~~ of ~~Hybrid Systems~~. A. ~~Donze (~~2010) [https://link.springer.com/content/pdf/10.1007/978-3-642-14295-6_17.pdf Link]		6. Donzé, Alexandre. "Breach, a toolbox for verification and parameter synthesis of hybrid systems." International Conference on Computer Aided Verification. Springer, Berlin, Heidelberg, 2010. [https://link.springer.com/content/pdf/10.1007/978-3-642-14295-6_17.pdf Link]

Abadithe at 02:02, 11 December 2019

2019-12-11T02:02:07Z

← Older revision		Revision as of 02:02, 11 December 2019
Line 18:		Line 18:
	3) Choose environment actions in test based on information gathered on system actions.		3) Choose environment actions in test based on information gathered on system actions.

	One useful reference is '''Breach''', a MATLAB-based monitoring tool that accepts STL specifications and the system state and output any violations of the specifications.		One useful reference is ''Breach'', a MATLAB-based monitoring tool that accepts STL specifications and the system state and output any violations of the specifications.

	It would be useful for the SURF student to know MATLAB and Python.		It would be useful for the SURF student to know MATLAB and Python.

Abadithe at 02:01, 11 December 2019

2019-12-11T02:01:35Z

← Older revision		Revision as of 02:01, 11 December 2019
Line 1:		Line 1:
	Autonomous systems are an emerging technology with potential for growth and impact in safety-critical applications such as self-driving cars, space missions, distributed power grid. In these applications, a rigorous, proof-based framework for design, test and evaluation of autonomy is necessary.		Autonomous systems are an emerging technology with potential for growth and impact in safety-critical applications such as self-driving cars, space missions, distributed power grid. In these applications, a rigorous, proof-based framework for design, test and evaluation of autonomy is necessary.

			'''Background'''<br>
	The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.		The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.

Line 8:		Line 9:
	If the system and environment specifications are given as GR(1) LTL specifications, it is possible to synthesize correct-by-construction controllers using TuliP (Reference 5). More background on GR(1) formalism can be found in Reference 4. In this work, we use GR(1) specifications to describe system and environment requirements.		If the system and environment specifications are given as GR(1) LTL specifications, it is possible to synthesize correct-by-construction controllers using TuliP (Reference 5). More background on GR(1) formalism can be found in Reference 4. In this work, we use GR(1) specifications to describe system and environment requirements.

	In testing, we check if the system has satisfied its specifications for a finite number of possible combinations of initial conditions, model parameters and environment actions. From test data, we can then evaluate whether the system has passed the test. The challenge is how do we algorithmically generate the "best" (finite) set of tests for an autonomous system given the specifications of the system and the environment? To this end, we would like the SURF student to develop a Test Plan Monitor for discrete decision-making behaviors.		'''Project Scope'''<br>
			In testing, we check if the system has satisfied its specifications for a finite number of possible combinations of initial conditions, model parameters and environment actions. From test data, we can then evaluate whether the system has passed the test. The challenge is how do we algorithmically generate the "best" (finite) set of tests for an autonomous system given the specifications of the system and the environment? To this end, we would like the SURF student to develop a Test Plan Monitor for discrete decision-making behaviors. The test-plan monitor will be used for the following purposes:

	1)~~How do we leverage test data to design~~ the ~~next set of tests? Specifically, for each trace~~ of system ~~actions, can we generate a set of environment traces to form the next round of tests?~~		1) Monitor the execution of the system. <br>

	2)~~Use~~ test ~~data to identify if~~ the ~~fault was caused because~~ of ~~information not captured in~~ the ~~discrete system model~~.		2) Validate the discretized system model (also known as system abstraction), and assumptions about the environment. <br>

			3) Choose environment actions in test based on information gathered on system actions.

			One useful reference is '''Breach''', a MATLAB-based monitoring tool that accepts STL specifications and the system state and output any violations of the specifications.

	It would be useful for the SURF student to know MATLAB and Python.		It would be useful for the SURF student to know MATLAB and Python.
Line 27:		Line 33:

	5. Wongpiromsarn, Tichakorn, et al. "TuLiP: a software toolbox for receding horizon temporal logic planning." Proceedings of the 14th international conference on Hybrid systems: computation and control. ACM, 2011. [https://user.eng.umd.edu/~mumu/files/wtoxm_HSCC2011.pdf Link]		5. Wongpiromsarn, Tichakorn, et al. "TuLiP: a software toolbox for receding horizon temporal logic planning." Proceedings of the 14th international conference on Hybrid systems: computation and control. ACM, 2011. [https://user.eng.umd.edu/~mumu/files/wtoxm_HSCC2011.pdf Link]

			6. Breach, A Toolbox for Verification and Parameter Synthesis of Hybrid Systems. A. Donze (2010) [https://link.springer.com/content/pdf/10.1007/978-3-642-14295-6_17.pdf Link]

Abadithe at 00:44, 11 December 2019

2019-12-11T00:44:14Z

← Older revision		Revision as of 00:44, 11 December 2019
Line 4:		Line 4:

	[[Image:architecture_auton_sys.png\|thumb\|400px\|left\|Figure 1: Architecture of autonomous systems]]		[[Image:architecture_auton_sys.png\|thumb\|400px\|left\|Figure 1: Architecture of autonomous systems]]
	[[Image:gridworld_sys_env.png\|thumb\|400px\|~~right~~\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]		[[Image:gridworld_sys_env.png\|thumb\|400px\|left\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]

	If the system and environment specifications are given as GR(1) LTL specifications, it is possible to synthesize correct-by-construction controllers using TuliP (Reference 5). More background on GR(1) formalism can be found in Reference 4. In this work, we use GR(1) specifications to describe system and environment requirements.		If the system and environment specifications are given as GR(1) LTL specifications, it is possible to synthesize correct-by-construction controllers using TuliP (Reference 5). More background on GR(1) formalism can be found in Reference 4. In this work, we use GR(1) specifications to describe system and environment requirements.

Abadithe at 00:43, 11 December 2019

2019-12-11T00:43:48Z

← Older revision		Revision as of 00:43, 11 December 2019
Line 6:		Line 6:
	[[Image:gridworld_sys_env.png\|thumb\|400px\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]		[[Image:gridworld_sys_env.png\|thumb\|400px\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]

	~~A test~~ is ~~a sequence of environmental inputs~~ to the system ~~with the objective~~ of ~~finding faults in the system~~. From test data, we can evaluate whether the system has passed the test. ~~One~~ of the ~~difficulties with testing autonomous systems is that under~~ the ~~same environmental conditions~~, the ~~system might choose~~ to ~~take different actions.~~		If the system and environment specifications are given as GR(1) LTL specifications, it is possible to synthesize correct-by-construction controllers using TuliP (Reference 5). More background on GR(1) formalism can be found in Reference 4. In this work, we use GR(1) specifications to describe system and environment requirements.
	~~Here are~~ a ~~few different possibilities~~ for ~~a SURF project:~~
			In testing, we check if the system has satisfied its specifications for a finite number of possible combinations of initial conditions, model parameters and environment actions. From test data, we can then evaluate whether the system has passed the test. The challenge is how do we algorithmically generate the "best" (finite) set of tests for an autonomous system given the specifications of the system and the environment? To this end, we would like the SURF student to develop a Test Plan Monitor for discrete decision-making behaviors.

	1)How do we leverage test data to design the next set of tests? Specifically, for each trace of system actions, can we generate a set of environment traces to form the next round of tests?		1)How do we leverage test data to design the next set of tests? Specifically, for each trace of system actions, can we generate a set of environment traces to form the next round of tests?
Line 23:		Line 24:
	3. On Signal Temporal Logic. A. Donze, 2014 [https://people.eecs.berkeley.edu/~sseshia/fmee/lectures/EECS294-98_Spring2014_STL_Lecture.pdf Link]		3. On Signal Temporal Logic. A. Donze, 2014 [https://people.eecs.berkeley.edu/~sseshia/fmee/lectures/EECS294-98_Spring2014_STL_Lecture.pdf Link]

	4. Wongpiromsarn, Tichakorn, et al. "TuLiP: a software toolbox for receding horizon temporal logic planning." Proceedings of the 14th international conference on Hybrid systems: computation and control. ACM, 2011. [https://user.eng.umd.edu/~mumu/files/wtoxm_HSCC2011.pdf Link]		4. RM Murray, U. Topcu, N. Wongpiromsarn. HYCON-EECI, 2013 [http://www.cds.caltech.edu/~murray/courses/eeci-sp13/L7_reactive-20Mar13.pdf Link]

			5. Wongpiromsarn, Tichakorn, et al. "TuLiP: a software toolbox for receding horizon temporal logic planning." Proceedings of the 14th international conference on Hybrid systems: computation and control. ACM, 2011. [https://user.eng.umd.edu/~mumu/files/wtoxm_HSCC2011.pdf Link]

Abadithe at 23:37, 10 December 2019

2019-12-10T23:37:21Z

← Older revision		Revision as of 23:37, 10 December 2019
Line 3:		Line 3:
	The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.		The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.

	[[Image:architecture_auton_sys.png\|~~frame~~\|400px\|Figure 1: Architecture of autonomous systems]]		[[Image:architecture_auton_sys.png\|thumb\|400px\|left\|Figure 1: Architecture of autonomous systems]]
	[[Image:gridworld_sys_env.png\|400px~~\|frame~~\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]		[[Image:gridworld_sys_env.png\|thumb\|400px\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]

	A test is a sequence of environmental inputs to the system with the objective of finding faults in the system. From test data, we can evaluate whether the system has passed the test. One of the difficulties with testing autonomous systems is that under the same environmental conditions, the system might choose to take different actions.		A test is a sequence of environmental inputs to the system with the objective of finding faults in the system. From test data, we can evaluate whether the system has passed the test. One of the difficulties with testing autonomous systems is that under the same environmental conditions, the system might choose to take different actions.

Abadithe at 23:34, 10 December 2019

2019-12-10T23:34:46Z

← Older revision		Revision as of 23:34, 10 December 2019
Line 3:		Line 3:
	The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.		The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.

	[[Image:architecture_auton_sys.png~~\|400px~~\|frame\|~~left~~\|Figure 1: Architecture of autonomous systems]]		[[Image:architecture_auton_sys.png\|frame\|400px\|Figure 1: Architecture of autonomous systems]]
	[[Image:gridworld_sys_env.png\|400px\|frame\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]		[[Image:gridworld_sys_env.png\|400px\|frame\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]

Abadithe at 23:33, 10 December 2019

2019-12-10T23:33:45Z

← Older revision		Revision as of 23:33, 10 December 2019
Line 3:		Line 3:
	The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.		The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.

	[[Image:architecture_auton_sys.png\|frame~~\|400px~~\|left\|Figure 1: Architecture of autonomous systems]]		[[Image:architecture_auton_sys.png\|400px\|frame\|left\|Figure 1: Architecture of autonomous systems]]
	[[Image:gridworld_sys_env.png\|frame~~\|400px~~\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]		[[Image:gridworld_sys_env.png\|400px\|frame\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]

	A test is a sequence of environmental inputs to the system with the objective of finding faults in the system. From test data, we can evaluate whether the system has passed the test. One of the difficulties with testing autonomous systems is that under the same environmental conditions, the system might choose to take different actions.		A test is a sequence of environmental inputs to the system with the objective of finding faults in the system. From test data, we can evaluate whether the system has passed the test. One of the difficulties with testing autonomous systems is that under the same environmental conditions, the system might choose to take different actions.

Abadithe at 23:33, 10 December 2019

2019-12-10T23:33:20Z

← Older revision		Revision as of 23:33, 10 December 2019
Line 3:		Line 3:
	The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.		The architecture of autonomous systems can be represented in a hierarchy of levels (see figure below) with a discrete decision-making layer at the top and low-level controllers at the bottom. In this project, we will be focusing on testing at the top layer, that is, testing for discrete event systems. The requirements of the system under consideration can be represented as specifications in a formal language. Most commonly in theory, specifications are written in temporal logic languages such as Linear Temporal Logic (LTL), Signal Temporal Logic (STL), Computational Tree Logic (CTL), etc (See references 2 and 3 for an overview on writing specifications in temporal logic). In addition to the system and its specifications, we have an environment for which corresponding specifications can be written. The environment constitutes of other agents modeled in the problem statement that do not constitute the system. See Figure 2 below for an example scenario of the system / environment.

	[[Image:architecture_auton_sys.png\|frame\|left~~\|400px~~\|Figure 1: Architecture of autonomous systems]]		[[Image:architecture_auton_sys.png\|frame\|400px\|left\|Figure 1: Architecture of autonomous systems]]
	[[Image:gridworld_sys_env.png\|frame\|right~~\|400px~~\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]		[[Image:gridworld_sys_env.png\|frame\|400px\|right\|Figure 2: Blue agent constitutes the system. It's objective is to always eventually go to GOAL when the park signal is ON, and always eventually go HOME when the ''park'' signal is OFF. With every step, its FUEL decreases by 1 unit. The blue agent must accomplish these objectives without running out of FUEL and without hitting the obstacle (red) patrolling agent. The red patrolling car and the ''park'' signal constitute the environment. The ''park'' signal can only take ON/OFF values and the red agent can only move along the red-transition lines on the grid.]]

	A test is a sequence of environmental inputs to the system with the objective of finding faults in the system. From test data, we can evaluate whether the system has passed the test. One of the difficulties with testing autonomous systems is that under the same environmental conditions, the system might choose to take different actions.		A test is a sequence of environmental inputs to the system with the objective of finding faults in the system. From test data, we can evaluate whether the system has passed the test. One of the difficulties with testing autonomous systems is that under the same environmental conditions, the system might choose to take different actions.