bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)

The current behaviour of yield expressions inside comprehensions  and
generator expressions is essentially an accident of implementation - it
arises implicitly from the way the compiler handles yield expressions inside
nested functions and generators.

Since the current behaviour wasn't deliberately designed, and is inherently
confusing, we're deprecating it, with no current plans to reintroduce it.
Instead, our advice will be to use a named nested generator definition
for cases where this behaviour is desired.

1 files changed, 53 insertions, 8 deletions

diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst
index 1cff8a52df9..fb92ad0f07c 100644
--- a/Doc/reference/expressions.rst
+++ b/Doc/reference/expressions.rst
@@ -183,8 +183,21 @@ by considering each of the :keyword:`for` or :keyword:`if` clauses a block,
 nesting from left to right, and evaluating the expression to produce an element
 each time the innermost block is reached.
 
-Note that the comprehension is executed in a separate scope, so names assigned
-to in the target list don't "leak" into the enclosing scope.
+However, aside from the iterable expression in the leftmost :keyword:`for` clause,
+the comprehension is executed in a separate implicitly nested scope. This ensures
+that names assigned to in the target list don't "leak" into the enclosing scope.
+
+The iterable expression in the leftmost :keyword:`for` clause is evaluated
+directly in the enclosing scope and then passed as an argument to the implictly
+nested scope. Subsequent :keyword:`for` clauses and any filter condition in the
+leftmost :keyword:`for` clause cannot be evaluated in the enclosing scope as
+they may depend on the values obtained from the leftmost iterable. For example:
+``[x*y for x in range(10) for y in range(x, x+10)]``.
+
+To ensure the comprehension always results in a container of the appropriate
+type, ``yield`` and ``yield from`` expressions are prohibited in the implicitly
+nested scope (in Python 3.7, such expressions emit :exc:`DeprecationWarning`
+when compiled, in Python 3.8+ they will emit :exc:`SyntaxError`).
 
 Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for`
 clause may be used to iterate over a :term:`asynchronous iterator`.
@@ -198,6 +211,13 @@ or :keyword:`await` expressions it is called an
 suspend the execution of the coroutine function in which it appears.
 See also :pep:`530`.
 
+.. versionadded:: 3.6
+   Asynchronous comprehensions were introduced.
+
+.. deprecated:: 3.7
+   ``yield`` and ``yield from`` deprecated in the implicitly nested scope.
+
+
 .. _lists:
 
 List displays
@@ -316,27 +336,42 @@ brackets or curly braces.
 
 Variables used in the generator expression are evaluated lazily when the
 :meth:`~generator.__next__` method is called for the generator object (in the same
-fashion as normal generators).  However, the leftmost :keyword:`for` clause is
-immediately evaluated, so that an error produced by it can be seen before any
-other possible error in the code that handles the generator expression.
-Subsequent :keyword:`for` clauses cannot be evaluated immediately since they
-may depend on the previous :keyword:`for` loop. For example: ``(x*y for x in
-range(10) for y in bar(x))``.
+fashion as normal generators).  However, the iterable expression in the
+leftmost :keyword:`for` clause is immediately evaluated, so that an error
+produced by it will be emitted at the point where the generator expression
+is defined, rather than at the point where the first value is retrieved.
+Subsequent :keyword:`for` clauses and any filter condition in the leftmost
+:keyword:`for` clause cannot be evaluated in the enclosing scope as they may
+depend on the values obtained from the leftmost iterable. For example:
+``(x*y for x in range(10) for y in range(x, x+10))``.
 
 The parentheses can be omitted on calls with only one argument.  See section
 :ref:`calls` for details.
 
+To avoid interfering with the expected operation of the generator expression
+itself, ``yield`` and ``yield from`` expressions are prohibited in the
+implicitly defined generator (in Python 3.7, such expressions emit
+:exc:`DeprecationWarning` when compiled, in Python 3.8+ they will emit
+:exc:`SyntaxError`).
+
 If a generator expression contains either :keyword:`async for`
 clauses or :keyword:`await` expressions it is called an
 :dfn:`asynchronous generator expression`.  An asynchronous generator
 expression returns a new asynchronous generator object,
 which is an asynchronous iterator (see :ref:`async-iterators`).
 
+.. versionadded:: 3.6
+   Asynchronous generator expressions were introduced.
+
 .. versionchanged:: 3.7
    Prior to Python 3.7, asynchronous generator expressions could
    only appear in :keyword:`async def` coroutines.  Starting
    with 3.7, any function can use asynchronous generator expressions.
 
+.. deprecated:: 3.7
+   ``yield`` and ``yield from`` deprecated in the implicitly nested scope.
+
+
 .. _yieldexpr:
 
 Yield expressions
@@ -364,6 +399,16 @@ coroutine function to be an asynchronous generator. For example::
     async def agen(): # defines an asynchronous generator function (PEP 525)
         yield 123
 
+Due to their side effects on the containing scope, ``yield`` expressions
+are not permitted as part of the implicitly defined scopes used to
+implement comprehensions and generator expressions (in Python 3.7, such
+expressions emit :exc:`DeprecationWarning` when compiled, in Python 3.8+
+they will emit :exc:`SyntaxError`)..
+
+.. deprecated:: 3.7
+   Yield expressions deprecated in the implicitly nested scopes used to
+   implement comprehensions and generator expressions.
+
 Generator functions are described below, while asynchronous generator
 functions are described separately in section
 :ref:`asynchronous-generator-functions`.