Parallel output paths for acoustic echo cancellation转让专利
申请号 : US15185799
文献号 : US09818425B1
文献日 : 2017-11-14
发明人 : Robert Ayrapetian , Philip Ryan Hilmes , Wai Chung Chu , Hyeong Cheol Kim , Yuwen Su
申请人 : Amazon Technologies, Inc.
摘要 :
权利要求 :
What is claimed is:
说明书 :
In audio systems, acoustic echo cancellation (AEC) refers to techniques that are used to recognize when a system has recaptured sound via a microphone after some delay that the system previously output via a speaker. Systems that provide AEC subtract a delayed version of the original audio signal from the captured audio, producing a version of the captured audio that ideally eliminates the “echo” of the original audio signal, leaving only new audio information. For example, if someone were singing karaoke into a microphone while prerecorded music is output by a loudspeaker, AEC can be used to remove any of the recorded music from the audio captured by the microphone, allowing the singer's voice to be amplified and output without also reproducing a delayed “echo” the original music. As another example, a media player that accepts voice commands via a microphone can use AEC to remove reproduced sounds corresponding to output media that are captured by the microphone, making it easier to process input voice commands.
For a more complete understanding of the present disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.
Typically, a conventional Acoustic Echo Cancellation (AEC) system may remove audio output by a loudspeaker from audio captured by the system's microphone(s) by subtracting a delayed version of the originally transmitted audio. However, in stereo and multi-channel audio systems that include wireless or network-connected loudspeakers and/or microphones, problem with the typical AEC approach may occur when there are differences between the signal sent to a loudspeaker and a signal received at the microphone. As the signal sent to the loudspeaker is not the same as the signal received at the microphone, the signal sent to the loudspeaker is not a true reference signal for the AEC system. For example, when the AEC system attempts to remove the audio output by the loudspeaker from audio captured by the system's microphone(s) by subtracting a delayed version of the originally transmitted audio, the audio captured by the microphone may be subtly different than the audio that had been sent to the loudspeaker.
To perform Acoustic Echo Cancellation (AEC) without knowing the signal played by the loudspeaker, an Adaptive Reference Signal Selection Algorithm (ARSSA) AEC system may perform audio beamforming on a signal received by the microphones and may determine a reference signal and a target signal based on the audio beamforming. For example, the ARSSA AEC system may receive audio input and separate the audio input into multiple directions. The ARSSA AEC system may detect a strong signal associated with a speaker and may set the strong signal as a reference signal, selecting another direction as a target signal. In some examples, the ARSSA AEC system may determine a speech position (e.g., near end talk position) and may set the direction associated with the speech position as a target signal and an opposite direction as a reference signal. If the ARSSA AEC system cannot detect a strong signal or determine a speech position, the system may create pairwise combinations of opposite directions, with an individual direction being used as a target signal and a reference signal. The ARSSA AEC system may remove the reference signal (e.g., audio output by the loudspeaker) to isolate speech included in the target signal.
To improve Automatic Speech Recognition (ASR), the AEC system may prioritize first speech from a single user and ignore second speech from additional users and/or environmental noise. Therefore, the AEC system may select audio associated with a single direction (e.g., speech position) as a target signal and remove reference signal(s) from the target signal to isolate the first speech. However, when there is speech from multiple users in multiple directions, isolating a single direction includes only a portion of the speech. In some situations, such as during a conference call, it is desirable to include all of the speech and/or environmental noise. Therefore, the AEC system may select audio associated with multiple directions as target signals, remove reference signal(s) from the target signals to isolate speech and/or environmental noise in individual directions and generate a combined output including portions from each of the individual directions. However, while the AEC system generates the combined output desirable for the conference call, ASR performance may be negatively impacted due to overlapping speech from multiple users.
To further improve echo cancellation, devices, systems and methods may combine the advantages of single direction AEC, which is desirable for ASR, and multi-direction AEC, which is desirable for conference calls. Thus, a device may be configured to provide two output paths to enable ASR processing in parallel with voice communication (e.g., Voice over Internet Protocol (VoIP)). For example, for single direction AEC (e.g., ASR processing), the device 102 may prioritize speech from a single user and ignore other speech and/or environmental noise by selecting a single directional output from a plurality of directional outputs. For multi-directional AEC (e.g., voice communication), the device may include all speech and environmental noise by combining the plurality of directional outputs. In some examples, the device may combine the plurality of directional outputs using a weighted sum technique, such that each directional output is represented in the combined output based on a corresponding signal metric (e.g., signal to noise ratio (SNR) or power) relative to the plurality of directional outputs. In other examples, the device may combine the plurality of directional outputs using an equal weighting technique, such that a first group of directional outputs having a higher signal metric may be equally weighted using a first weight while a second group of directional outputs having a lower signal metric may be equally weighted using a second weight.
For ASR processing, the device 102 may prioritize speech from a single user and ignore other speech and/or environmental noise. Thus, the device 102 may generate the first audio output 128a by selecting a single output from the plurality of outputs. In contrast, for VoIP communication, the device 102 may include all speech and environmental noise. Thus, the device 102 may generate the second audio output 128b by combining the plurality of outputs. The device 102 may generate the second audio output 128b using a weighted sum technique or an equal weighting technique, as will be discussed in greater detail below.
As illustrated in
The device 102 may include an adaptive beamformer 104 that may perform audio beamforming on the echo signals 120 to determine target signals 122 (e.g., 122a, 122b . . . 122m). For example, the adaptive beamformer 104 may include a fixed beamformer (FBF) 105, a multiple input canceler (MC) 106 and/or a blocking matrix (BM) 107. The FBF 105 may be configured to form a beam in a specific direction so that a target signal is passed and all other signals are attenuated, enabling the adaptive beamformer 104 to select a particular direction. In contrast, the BM 107 may be configured to form a null in a specific direction so that the target signal is attenuated and all other signals are passed. The adaptive beamformer 104 may generate fixed beamforms (e.g., outputs of the FBF 105) or may generate adaptive beamforms using a Linearly Constrained Minimum Variance (LCMV) beamformer, a Minimum Variance Distortionless Response (MVDR) beamformer or other beamforming techniques. For example, the adaptive beamformer 104 may receive audio input, determine six beamforming directions and output six fixed beamform outputs and six adaptive beamform outputs. In some examples, the adaptive beamformer 104 may generate six fixed beamform outputs, six LCMV beamform outputs and six MVDR beamform outputs, although the disclosure is not limited thereto.
In some examples, the adaptive beamformer 104 may generate the same number of outputs as inputs, such that there is an equal number of echo signals 120 and target signals 122. However, the disclosure is not limited thereto and the number of target signals 122 may vary and does not need to match the number of echo signals 120.
While
Using the adaptive beamformer 104 and techniques discussed below, the device 102 may determine the target signals 122 to pass to an acoustic echo cancellation (AEC) 108. Additionally, the adaptive beamformer 104 may use an Adaptive Reference Signal Selection Algorithm (ARSSA) to generate an adaptive reference signal 123. However, the disclosure is not limited thereto and the device 102 may generate a reference signal using techniques known to one of skill in the art. For example, the device 102 may include playback reference logic (not shown) that may generate a playback reference signal instead of the adaptive reference signal 123. To generate the playback reference signal, the playback reference logic may receive the reference signals 112 (e.g., originally transmitted audio) and may compensate for distortion, variable delay, drift, skew and/or frequency offset. For example, the playback reference logic may determine a propagation delay between the reference signals 112 and the echo signals 120 and may modify the reference signals 112 to remove the propagation delay. Additionally or alternatively, the playback reference logic may determine a frequency offset between the modified reference signals 112 and the echo signals 120 and may add/drop samples of the modified reference signals and/or the echo signals 120 to compensate for the frequency offset. For example, the playback reference logic may add at least one sample per cycle when the frequency offset is positive and may remove at least one sample per cycle when the frequency offset is negative. Therefore, the playback reference signal may be aligned with the echo signals 120.
The device 102 may determine the target signals 122 and the adaptive reference signal 123 using multiple techniques. For example, the device 102 may use a first technique when the device 102 detects a clearly defined speaker signal, a second technique when the device 102 doesn't detect a clearly defined speaker signal but does identify a speech position and/or a third technique when the device 102 doesn't detect a clearly defined speaker signal or a speech position. Using the first technique, the device 102 may associate the clearly defined speaker signal with the adaptive reference signal 123 and may select any or all of the other directions as the target signal(s) 122. For example, the device 102 may generate a single target signal 122 using all of the remaining directions for a single loudspeaker or may generate multiple target signals 122 using portions of remaining directions for multiple loudspeakers. Using the second technique, the device 102 may associate the speech position with the target signal 122 and may select an opposite direction as the adaptive reference signal 123. Using the third technique, the device 102 may select multiple combinations of opposing directions to generate multiple target signals 122 and multiple adaptive reference signals 123.
In some examples, the device 102 may associate specific directions with the reproduced sounds and/or speech based on features of the signal sent to the loudspeaker. Examples of features includes power spectrum density, peak levels, pause intervals or the like that may be used to identify the signal sent to the loudspeaker and/or propagation delay between different signals. For example, the adaptive beamformer 104 may compare the signal sent to the loudspeaker with a signal associated with a first direction to determine if the signal associated with the first direction includes reproduced sounds from the loudspeaker. When the signal associated with the first direction matches the signal sent to the loudspeaker, the device 102 may associate the first direction with a wireless speaker 114 (e.g., use the first direction as the adaptive reference signal 123). When the signal associated with the first direction does not match the signal sent to the loudspeaker, the device 102 may associate the first direction with speech, a speech position, a person or the like. In some examples, the adaptive beamformer 104 may generate the target signals 122 using pairwise combinations of different directions.
The device 102 may optionally determine a speech position (e.g., near end talk position) associated with speech and/or a person speaking. For example, the device 102 may identify the speech, a person and/or a position associated with the speech/person using audio data (e.g., audio beamforming when speech is recognized), video data (e.g., facial recognition) and/or other inputs known to one of skill in the art. The device 102 may determine target signals 122, which may include a single target signal (e.g., echo signal 120 received from a microphone 118) or may include multiple target signals (e.g., target signal 122a, target signal 122b, . . . target signal 122n) that may be generated using the FBF 105 or other components of the adaptive beamformer 104. In some examples, the device 102 may determine the target signals based on the speech position. The device 102 may determine an adaptive reference signal based on the speech position and/or the audio beamforming. For example, the device 102 may associate the speech position with a target signal and may select an opposite direction as the adaptive reference signal.
The AEC 108 may remove the adaptive reference signal 123 (e.g., reproduced sounds) from the target signals 122 (e.g., reproduced sounds and additional sounds) to remove the reproduced sounds and isolate the additional sounds (e.g., speech) as directional outputs 124 (e.g., 124a, 124b . . . 124m).
To illustrate, in some examples the device 102 may use outputs of the FBF 105 as the target signals 122. For example, the outputs of the FBF 105 may be shown in equation 1:
Target Signal=s+z+noise (1)
where s is speech (e.g., the additional sounds), z is an echo from the signal sent to the loudspeaker (e.g., the reproduced sounds) and noise is additional noise that is not associated with the speech or the echo. In order to attenuate the echo (z), the device 102 may use the adaptive reference signal 123, which may be shown in equation 2:
Reference Signal=z+noise (2)
By removing the adaptive reference signal 123 from the target signals 122, the device 102 may remove the echo and generate the directional outputs 124 including only the speech and some noise.
The AEC 108 may generate a plurality of directional outputs 124 (e.g., 124a, 124b . . . 124m). For example, each of the plurality of directional outputs 124 may correspond to a separate target signal 122. Additionally or alternatively, each of the plurality of directional outputs 124 may correspond to a direction determined by the adaptive beamformer 104. While
The device 102 may include an output generator 126 that may receive the directional outputs and generate audio outputs 128. For example, the output generator 126 may generate a first audio output 128a (e.g., ASR output path) and a second audio output 128b (e.g., VoIP output path). As discussed above, to generate the first audio output 128a, the output generator 126 may prioritize speech from a single user and ignore other speech and/or environmental noise. Thus, the output generator 126 may determine a single directional output having a strongest signal of the directional outputs 124 and may select the single directional output as the first audio output 128a. In contrast, to generate the second audio output 128b, the output generator 126 include all speech and environmental noise. Thus, the output generator 126 may generate the second audio output 128b by combining the directional outputs 124.
As a first example, the output generator 126 may generate the second audio output 128b using a weighted sum technique. For example, the output generator 126 may determine a signal strength for each of the directional outputs 124 and may generate a weight corresponding to the signal strength. Thus, a first directional output 124a may have a first weight, a second directional output 124b may have a second weight, and so on until an mth directional output 124m has an mth weight. The output generator 126 may generate the second audio output 128b by multiplying the directional outputs 124 by corresponding weights (e.g., first directional output 124a multiplied by the first weight, second directional output 124b multiplied by the second weight, etc.) and summing the products.
As a second example, the output generator 126 may generate the second audio output 128b using an equal weighting technique. For example, the output generator 126 may determine a signal strength for each of the directional outputs 124 and may separate the directional outputs 124 into two separate groups. Thus, if there are six directional outputs 124, the output generator 126 may select three directional outputs having a highest signal strength (e.g., 124a, 124b, 124c) as a first group and select three directional outputs having a lowest signal strength (e.g., 124d, 124e, 124f) as a second group. After separating the six directional outputs into the two groups, the output generator 126 may assign a first weight (e.g., higher priority) to the first group and a second weight (e.g., lower priority) to the second group. For example, the first group may be assigned a first weight so that the sum of the first weights is equal to a first percentage (e.g., 70%), and the second group may be assigned a second weight so that the sum of the second weights is equal to a second percentage (e.g., 30%). In the example discussed above, the first weight would be approximately 0.233 (for a total sum of 0.7), whereas the second weight would be approximately 0.1 (for a total sum of 0.3). The output generator 126 may generate the second audio output 128b by multiplying the directional outputs 124 by the corresponding weights (e.g., directional output 124a-124c multiplied by the first weight, directional outputs 124d-124f multiplied by the second weight) and summing the products.
The audio outputs 128 may be processed by the device 102, sent to a remote device for further processing and/or sent to a remote device during voice communication. For example, the second audio output 128b may be sent to a remote device as part of VoIP communication. Additionally or alternatively, the device 102 (or a remote device, such as a server) may perform speech recognition processing on the first audio output 128a to determine a command and may execute the command. For example, the device 102 may determine that the speech corresponds to a command to play music and the device 102 may play music in response to receiving the speech. The device 102 may perform additional actions based on other commands, such as changing a volume of audio output by the device 102 and/or wireless speakers 114, launching an application, ending the VoIP communication or the like. In some examples, the device 102 may send the first audio output 128a to a server, the server may perform speech processing on the first audio output 128a to determine the command and may send the command to the device 102 and the device 102 may execute the command.
As illustrated in
The device 102 may determine (132) signal metrics, such as signal to noise ratios (SNRs) associated with the directional outputs 124. For example, the device 102 may determine a first SNR associated with the first directional output 124a, a second SNR associated with a second directional output 124b, etc. However, SNR is just a single example of a parameter or characteristic associated with the directional outputs 124 and the present disclosure is not limited thereto. Instead of using SNRs, the device 102 may compare the directional outputs 124 using other characteristics or parameters (e.g., signal strength, power, etc.).
The device 102 may determine (134) weights associated with the directional outputs 124 based on the signal metrics. For example, the device 102 may use the first SNR to determine a first weight corresponding to the first directional output 124a, use the second SNR to determine a second weight corresponding to the second directional output 124b, and so on. As discussed above, the signal metrics are not limited to SNRs and the device 102 may determine the weights using any signal metrics known to one of skill in the art. For example, the device 102 may determine the first weight using a first magnitude of power associated with the first directional output 124a without departing from the disclosure.
As discussed above, the device 102 may determine the weights using multiple techniques, including a weighted sum technique and/or an equal weighting technique. To illustrate the weighted sum technique, a first directional output 124a may have a first weight, a second directional output 124b may have a second weight, and so on until an mth directional output 124m has an mth weight. To illustrate the equal weighting technique, a first group of directional outputs 124 (e.g., 124a, 124b and 124c) may have a first weight (e.g., 0.233) and a second group of directional outputs 124 (e.g., 124d, 124e and 124f) may have a second weight (e.g., 0.1).
The device 102 may select (136) a single directional output as a first audio output 128a. For example, the device 102 may determine a first directional output 124a having a highest SNR and may select the first directional output 124a as the first audio output 128a.
The device 102 may generate (138) a second audio output 128b using the directional outputs and the weights. Using the weighted sum example, the device 102 may generate the second audio output 128b by multiplying the directional outputs 124 by corresponding weights (e.g., first directional output 124a multiplied by the first weight, second directional output 124b multiplied by the second weight, etc.) and summing the products. Using the equal weighting example, the device 102 may generate the second audio output 128b by multiplying the directional outputs 124 by corresponding weights (e.g., directional output 124a-124c multiplied by the first weight, directional outputs 124d-124f multiplied by the second weight) and summing the products.
The device 102 may include a microphone array having multiple microphones 118 that are laterally spaced from each other so that they can be used by audio beamforming components to produce directional audio signals. The microphones 118 may, in some instances, be dispersed around a perimeter of the device 102 in order to apply beampatterns to audio signals based on sound captured by the microphone(s) 118. For example, the microphones 118 may be positioned at spaced intervals along a perimeter of the device 102, although the present disclosure is not limited thereto. In some examples, the microphone(s) 118 may be spaced on a substantially vertical surface of the device 102 and/or a top surface of the device 102. Each of the microphones 118 is omnidirectional, and beamforming technology is used to produce directional audio signals based on signals from the microphones 118. In other embodiments, the microphones may have directional audio reception, which may remove the need for subsequent beamforming.
In various embodiments, the microphone array may include greater or less than the number of microphones 118 shown. Speaker(s) (not illustrated) may be located at the bottom of the device 102, and may be configured to emit sound omnidirectionally, in a 360 degree pattern around the device 102. For example, the speaker(s) may comprise a round speaker element directed downwardly in the lower part of the device 102.
Using the plurality of microphones 118 the device 102 may employ beamforming techniques to isolate desired sounds for purposes of converting those sounds into audio signals for speech processing by the system. Beamforming is the process of applying a set of beamformer coefficients to audio signal data to create beampatterns, or effective directions of gain or attenuation. In some implementations, these volumes may be considered to result from constructive and destructive interference between signals from individual microphones in a microphone array.
The device 102 may include an adaptive beamformer 104 that may include one or more audio beamformers or beamforming components that are configured to generate an audio signal that is focused in a direction from which user speech has been detected. More specifically, the beamforming components may be responsive to spatially separated microphone elements of the microphone array to produce directional audio signals that emphasize sounds originating from different directions relative to the device 102, and to select and output one of the audio signals that is most likely to contain user speech.
Audio beamforming, also referred to as audio array processing, uses a microphone array having multiple microphones that are spaced from each other at known distances. Sound originating from a source is received by each of the microphones. However, because each microphone is potentially at a different distance from the sound source, a propagating sound wave arrives at each of the microphones at slightly different times. This difference in arrival time results in phase differences between audio signals produced by the microphones. The phase differences can be exploited to enhance sounds originating from chosen directions relative to the microphone array.
Beamforming uses signal processing techniques to combine signals from the different microphones so that sound signals originating from a particular direction are emphasized while sound signals from other directions are deemphasized. More specifically, signals from the different microphones are combined in such a way that signals from a particular direction experience constructive interference, while signals from other directions experience destructive interference. The parameters used in beamforming may be varied to dynamically select different directions, even when using a fixed-configuration microphone array.
A given beampattern may be used to selectively gather signals from a particular spatial location where a signal source is present. The selected beampattern may be configured to provide gain or attenuation for the signal source. For example, the beampattern may be focused on a particular user's head allowing for the recovery of the user's speech while attenuating noise from an operating air conditioner that is across the room and in a different direction than the user relative to a device that captures the audio signals.
Such spatial selectivity by using beamforming allows for the rejection or attenuation of undesired signals outside of the beampattern. The increased selectivity of the beampattern improves signal-to-noise ratio for the audio signal. By improving the signal-to-noise ratio, the accuracy of speaker recognition performed on the audio signal is improved.
The processed data from the beamformer module may then undergo additional filtering or be used directly by other modules. For example, a filter may be applied to processed data which is acquiring speech from a user to remove residual audio noise from a machine running in the environment.
The beampattern 202 may exhibit a plurality of lobes, or regions of gain, with gain predominating in a particular direction designated the beampattern direction 204. A main lobe 206 is shown here extending along the beampattern direction 204. A main lobe beam-width 208 is shown, indicating a maximum width of the main lobe 206. In this example, the beampattern 202 also includes side lobes 210, 212, 214, and 216. Opposite the main lobe 206 along the beampattern direction 204 is the back lobe 218. Disposed around the beampattern 202 are null regions 220. These null regions are areas of attenuation to signals. In the example, the person 10 resides within the main lobe 206 and benefits from the gain provided by the beampattern 202 and exhibits an improved SNR ratio compared to a signal acquired with non-beamforming. In contrast, if the person 10 were to speak from a null region, the resulting audio signal may be significantly reduced. As shown in this illustration, the use of the beampattern provides for gain in signal acquisition compared to non-beamforming. Beamforming also allows for spatial selectivity, effectively allowing the system to “turn a deaf ear” on a signal which is not of interest. Beamforming may result in directional audio signal(s) that may then be processed by other components of the device 102 and/or system 100.
While beamforming alone may increase a signal-to-noise (SNR) ratio of an audio signal, combining known acoustic characteristics of an environment (e.g., a room impulse response (RIR)) and heuristic knowledge of previous beampattern lobe selection may provide an even better indication of a speaking user's likely location within the environment. In some instances, a device includes multiple microphones that capture audio signals that include user speech. As is known and as used herein, “capturing” an audio signal includes a microphone transducing audio waves of captured sound to an electrical signal and a codec digitizing the signal. The device may also include functionality for applying different beampatterns to the captured audio signals, with each beampattern having multiple lobes. By identifying lobes most likely to contain user speech using the combination discussed above, the techniques enable devotion of additional processing resources of the portion of an audio signal most likely to contain user speech to provide better echo canceling and thus a cleaner SNR ratio in the resulting processed audio signal.
To determine a value of an acoustic characteristic of an environment (e.g., an RIR of the environment), the device 102 may emit sounds at known frequencies (e.g., chirps, text-to-speech audio, music or spoken word content playback, etc.) to measure a reverberant signature of the environment to generate an RIR of the environment. Measured over time in an ongoing fashion, the device may be able to generate a consistent picture of the RIR and the reverberant qualities of the environment, thus better enabling the device to determine or approximate where it is located in relation to walls or corners of the environment (assuming the device is stationary). Further, if the device is moved, the device may be able to determine this change by noticing a change in the RIR pattern. In conjunction with this information, by tracking which lobe of a beampattern the device most often selects as having the strongest spoken signal path over time, the device may begin to notice patterns in which lobes are selected. If a certain set of lobes (or microphones) is selected, the device can heuristically determine the user's typical speaking location in the environment. The device may devote more CPU resources to digital signal processing (DSP) techniques for that lobe or set of lobes. For example, the device may run acoustic echo cancelation (AEC) at full strength across the three most commonly targeted lobes, instead of picking a single lobe to run AEC at full strength. The techniques may thus improve subsequent automatic speech recognition (ASR) and/or speaker recognition results as long as the device is not rotated or moved. And, if the device is moved, the techniques may help the device to determine this change by comparing current RIR results to historical ones to recognize differences that are significant enough to cause the device to begin processing the signal coming from all lobes approximately equally, rather than focusing only on the most commonly targeted lobes.
By focusing processing resources on a portion of an audio signal most likely to include user speech, the SNR of that portion may be increased as compared to the SNR if processing resources were spread out equally to the entire audio signal. This higher SNR for the most pertinent portion of the audio signal may increase the efficacy of the device 102 when performing speaker recognition on the resulting audio signal.
Using the beamforming and directional based techniques above, the system may determine a direction of detected audio relative to the audio capture components. Such direction information may be used to link speech/a recognized speaker identity to video data as described below.
As illustrated in
The number of portions/sections generated using beamforming does not depend on the number of microphones in the microphone array. For example, the device 102 may include twelve microphones in the microphone array but may determine three portions, six portions or twelve portions of the audio data without departing from the disclosure. While the examples illustrated in
As discussed above, the adaptive beamformer 104 may generate fixed beamforms (e.g., outputs of the FBF 105) or may generate adaptive beamforms using a Linearly Constrained Minimum Variance (LCMV) beamformer, a Minimum Variance Distortionless Response (MVDR) beamformer or other beamforming techniques. For example, the adaptive beamformer 104 may receive the audio input, may determine six beamforming directions and output six fixed beamform outputs and six adaptive beamform outputs corresponding to the six beamforming directions. In some examples, the adaptive beamformer 104 may generate six fixed beamform outputs, six LCMV beamform outputs and six MVDR beamform outputs, although the disclosure is not limited thereto.
The device 102 may determine a number of wireless loudspeakers and/or directions associated with the wireless loudspeakers using the fixed beamform outputs. For example, the device 102 may localize energy in the frequency domain and clearly identify much higher energy in two directions associated with two wireless loudspeakers (e.g., a first direction associated with a first speaker and a second direction associated with a second speaker). In some examples, the device 102 may determine an existence and/or location associated with the wireless loudspeakers using a frequency range (e.g., 1 kHz to 3 kHz), although the disclosure is not limited thereto. In some examples, the device 102 may determine an existence and location of the wireless speaker(s) using the fixed beamform outputs, may select a portion of the fixed beamform outputs as the target signal(s) and may select a portion of adaptive beamform outputs corresponding to the wireless speaker(s) as the reference signal(s).
To perform echo cancellation, the device 102 may determine a target signal and a reference signal and may remove the reference signal from the target signal to generate a directional output 124. For example, the loudspeaker may output audible sound associated with a first direction and a person may generate speech associated with a second direction. To remove the audible sound output from the loudspeaker, the device 102 may select a first portion of audio data corresponding to the first direction as the reference signal and may select a second portion of the audio data corresponding to the second direction as the target signal. However, the disclosure is not limited to a single portion being associated with the reference signal and/or target signal and the device 102 may select multiple portions of the audio data corresponding to multiple directions as the reference signal/target signal without departing from the disclosure. For example, the device 102 may select a first portion and a second portion as the reference signal and may select a third portion and a fourth portion as the target signal.
Additionally or alternatively, the device 102 may determine more than one reference signal and/or target signal. For example, the device 102 may identify a first wireless speaker and a second wireless speaker and may determine a first reference signal associated with the first wireless speaker and determine a second reference signal associated with the second wireless speaker. The device 102 may generate a first directional output 124 by removing the first reference signal from the target signal and may generate a second directional output 124 by removing the second reference signal from the target signal. Similarly, the device 102 may select a first portion of the audio data as a first target signal and may select a second portion of the audio data as a second target signal. The device 102 may therefore generate a first directional output 124 by removing the reference signal from the first target signal and may generate a second directional output 124 by removing the reference signal from the second target signal.
The device 102 may determine reference signals, target signals and/or directional output signals using any combination of portions of the audio data without departing from the disclosure. For example, the device 102 may select first and second portions of the audio data as a first reference signal, may select a third portion of the audio data as a second reference signal and may select remaining portions of the audio data as a target signal. In some examples, the device 102 may include the first portion in a first reference signal and a second reference signal or may include the second portion in a first target signal and a second target signal. If the device 102 selects multiple target signals and/or reference signals, the device 102 may remove each reference signal from each of the target signals individually (e.g., remove reference signal 1 from target signal 1, remove reference signal 1 from target signal 2, remove reference signal 2 from target signal 1, etc.), may collectively remove the reference signals from each individual target signal (e.g., remove reference signals 1-2 from target signal 1, remove reference signals 1-2 from target signal 2, etc.), remove individual reference signals from the target signals collectively (e.g., remove reference signal 1 from target signals 1-2, remove reference signal 2 from target signals 1-2, etc.) or any combination thereof without departing from the disclosure.
The device 102 may select fixed beamform outputs or adaptive beamform outputs as the target signal(s) and/or the reference signal(s) without departing from the disclosure. In a first example, the device 102 may select a first fixed beamform output (e.g., first portion of the audio data determined using fixed beamforming techniques) as a reference signal and a second fixed beamform output as a target signal. In a second example, the device 102 may select a first adaptive beamform output (e.g., first portion of the audio data determined using adaptive beamforming techniques) as a reference signal and a second adaptive beamform output as a target signal. In a third example, the device 102 may select the first fixed beamform output as the reference signal and the second adaptive beamform output as the target signal. In a fourth example, the device 102 may select the first adaptive beamform output as the reference signal and the second fixed beamform output as the target signal. However, the disclosure is not limited thereto and further combinations thereof may be selected without departing from the disclosure.
The device 102 may generate directional outputs 124 using the selected target signal(s) and reference signal(s). In some examples, the device 102 may generate a directional output 124 corresponding to each of the sections in a beamforming configuration. For example, the beamforming configuration 310 illustrated in
The device 102 may determine the directional outputs 124 using different techniques. For example, the device 102 may use a first technique when the device 102 detects a clearly defined speaker signal (e.g., wireless speaker 114), a second technique when the device 102 doesn't detect a clearly defined speaker signal but does identify a speech position (e.g., near end talk position) associated with a person, and/or a third technique when the device 102 doesn't detect a clearly defined speaker signal or a speech position.
To illustrate the first technique, the device 102 may detect a clearly defined speaker signal. For example, the device 102 may detect a wireless speaker 114 in Section 1 of the beamforming configuration 310 and the device 102 may associate the wireless speaker 114 with Section 1. To generate directional outputs 124, the device 102 may set Section 1 as a reference signal and may identify one or more remaining sections as target signal(s). For example, the device 102 may generate a plurality of directional outputs 124, with each directional output 124 corresponding to a single section (e.g., Section 2, Section 3, etc.) using Section 1 as the reference signal. Additionally or alternatively, the device 102 may generate one or more directional outputs 124 using a combination of sections as the target signals. If the device 102 detects an additional wireless speaker 114 in Section 6, the device 102 may generate directional outputs 124 using any combination of the remaining sections (e.g., Sections 2-5) as target signals and Section 1, Section 6 or a combination thereof as reference signal(s). For example, the device 102 may generate first directional outputs 124 using Section 1 as a reference signal, second directional outputs 124 using Section 6 as a reference signal and/or third directional outputs 124 using Sections 1 and 6 as reference signals.
To illustrate the second technique, the device 102 may not detect a clearly defined speaker signal but may identify a speech position (e.g., near end talk position) associated with a person. For example, the device 102 may identify the person and/or a position associated with the person using audio data (e.g., audio beamforming), video data (e.g., facial recognition) and/or other inputs known to one of skill in the art. In some examples, the device 102 may determine that speech is associated with a section and may determine a speech position using the section. In other examples, the device 102 may receive video data associated with the audio data and may use facial recognition or other techniques to determine a position associated with a face recognized in the video data. If the device 102 detects the person in Section 2, the device 102 may associate Section 2 with the person and use Section 2 as a target signal with one or more sections as reference signal(s). For example, the device 102 may generate directional outputs 124 using Section 2 as the target signal, with each directional output 124 corresponding to a single section (e.g., Section 1, Section 3, Section 4, etc.) as a reference signal. Additionally or alternatively, the device 102 may generate one or more directional outputs 124 using a combination of sections as the reference signals. If the device 102 identifies an additional speech position (e.g., second person) in Section 3, the device 102 may generate directional outputs 124 using any combination of the remaining sections (e.g., Section 1, Sections 4-6) as reference signals and Section 2, Section 3 or a combination thereof as target signal(s). For example, the device 102 may generate first directional outputs 124 using Section 2 as a target signal, second directional outputs 124 using Section 3 as a target signal and/or third directional outputs 124 using Sections 2 and 3 as target signals.
To illustrate the third technique, the device 102 may not detect a clearly defined speaker signal or identify a speech position (e.g., near end talk position) associated with a person. For example, audio from a wireless speaker 114 may reflect off of multiple objects such that the device 102 receives the audio from multiple locations at a time and is therefore unable to locate a specific section to associate with the wireless speaker. In this situation, the device 102 may generate directional outputs 124 using pairwise combinations of the sections illustrated in
After generating the first audio output 128a, the device 102 may perform Automatic Speech Recognition (ASR) on the first audio output 128a, may determine a command using the ASR output and may execute the command. For example, the device 102 may determine that speech included in the first audio output 128a corresponds to a command to play music and the device 102 may play music in response to receiving the speech. Additionally or alternatively, the device 102 may send the first audio output 128a to a remote device (e.g., server) to perform ASR and may receive the command from the remote device.
To illustrate an example, the device 102 may receive remote voice data corresponding to a remote person from a remote device (e.g., during VoIP communication) and may generate audio using the remote voice data, either using an internal loudspeaker or a wireless speaker 114. While the device 102 is generating the audio, the device 102 may capture local voice data corresponding to local people using a microphone 118. The device 102 may perform AEC on the local voice data to remove reproduced sounds corresponding to the remote voice data that are captured by the microphone 118 and may generate the first audio output 128a and the second audio output 128b. Using the first audio output 128a, the device 102 may determine and execute a command, enabling a local person to control the device 102 using voice commands (e.g., speech commands or the like). The device 102 may send the second audio output 128b to the remote device for playback.
As illustrated in
As the first audio output 128a is used to perform ASR to determine a voice command, the ASR processing is improved if the output generator 126 selects a single directional output for an entirety of the voice command. For example, a first person may input a first voice command by speaking for ten seconds. If a second person interrupts and inputs a second voice command after five seconds, the output generator 126 may prematurely switch from the first voice command to the second voice command, resulting in an incomplete and/or incomprehensible voice command. For example, the output generator 126 may select a first directional output 124a (corresponding to the first voice command) for the first five seconds (e.g., 0-5 seconds) and a second directional output 124b (corresponding to the second voice command) for the second five seconds (e.g., 6-10 seconds), such that the first audio output 128a includes portions of the first voice command and the second voice command.
To improve the first audio output 128a, the output generator 126 may perform a variety of techniques to include an entirety of a voice command. As a first example, the output generator 126 may select a fixed speech position from which to input voice commands to the device 102. For example, the device 102 may determine that the first person is in charge of the device 102, determine a speech position of the first person, determine a directional output 124 corresponding to the speech position and select the directional output 124 as the first audio output 128a. Thus, the device 102 may associate the first person with the first directional output 124a and select the first directional output 124a as the first audio output 128a regardless of the signal metrics for the other directional outputs 124 (e.g., ignore the second directional output 124b when the second person is inputting the second voice command). As a second example, the output generator 126 may select varying speech positions based on a single person in charge of the device 102. For example, the device 102 may determine that the first person is in charge of the device 102, determine a current speech position of the first person, determine a directional output 124 corresponding to the current speech position and select the directional output 124 as the first audio output 128a. Thus, as the first person moves relative to the device 102, the device 102 may select a corresponding directional input 124 as the first audio output 128a. The device 102 may determine the current speech position using voice activity detection, voice recognition, facial recognition (if the device 102 has access to image data), or the like.
As a third example, the output generator 126 may maintain a directional input 124 for at least a fixed period of time. For example, the output generator 126 may determine that an SNR value of the second directional input 124b exceeds an SNR value of the first directional input 124a but may continue to output the first directional input 124a for the fixed period of time (e.g., 2 seconds, although the disclosure is not limited thereto and the fixed period of time may vary). As a fourth example, the output generator 126 may maintain a directional input 124 until the output generator 126 receives an indication that the voice command is complete. For example, the device 102 (or a remote device) may determine an endpoint (e.g., end of the first voice command) associated with the first directional input 124a using techniques known to one of skill in the art and the output generator 126 may maintain the first directional input 124a until the endpoint, at which point the output generator 126 may select the second directional input 124b as the first audio output 128a. Additionally or alternatively, the device 102 (or a remote device) may determine that the first directional output 124a corresponds to a complete command (e.g., a confidence level of a voice command generated by the ASR processing is above a threshold) and may maintain the first directional input 124a until the command is completed, at which point the output generator 126 may select the second directional input 124b as the first audio output 128a.
As illustrated in
In the weighted sum technique, the device 102 may determine a combined output Sout, normalized weights βi, first sum αsum, and raw weights αi with the following equations:
Sout=β1*S1+β2*S2+ . . . +βm*Sm [3]
βi=αi/αsum [4]
αsum=α1+α2+ . . . +αm [5]
αi=γi*SNRi [6.1]
(or)
αi=γi*Power_Beami [6.2]
γi=1/(1+exp(−SNRi/THR)) [7]
where i is a natural number corresponding to the number of directional outputs 124 (e.g., i=1, 2, m), SNRi is a magnitude of a signal to noise ratio (SNR) for the ith directional output, Power_Beami is a magnitude of power for the ith directional output, THR is a threshold value (e.g., 15 dB), γi is a scale factor, and a second sum βsum of the normalized weights βi is equal to a value of one:
βsum=β1+β2+ . . . +βm=1 [8]
In some examples, the device 102 may set r=1 to simplify a processing of the combined output Sout. Additionally or alternatively, the device 102 may use double talk detection (DTD) and/or voice activity detection (VAD) to prevent the device 102 from changing AEC coefficients. For example, the device 102 may use DTD and/or VAD to detect near end talk, and when near end talk is detected, the device 102 may freeze AEC coefficients. Thus, the device 102 may not update the AEC coefficients when a far end talker is silent, or if both a far end talker and a near end talker are active.
For example, raw weight chart 510 illustrates that a magnitude of the first raw weight α1 is much larger than a magnitude of the second raw weight α2, and the magnitude of the second raw weight α2 is much larger than a magnitude of the third raw weight α3. Raw weights 512 illustrates a first sum (e.g., αsum) of the raw weights α1-α3 along with a proportion of each raw weight α1-α3 relative to the first sum αsum. To determine the weighted sum of the directional outputs 124a-124c, the device 102 may normalize the raw weights α1-α3 by dividing the raw weights α1-α3 by the first sum αsum. For example, normalized weights 514 illustrates a second sum (e.g., βsum) of the normalized weights β1-β3, with the second sum βsum equal to a value of one. Thus, the device 102 determines the first normalized weight β1 by dividing the first raw weight α1 by the first sum αsum, the second normalized weight β2 by dividing the second raw weight α2 by the first sum αsum, and the third normalized weight β3 by dividing the third raw weight α3 by the first sum αsum. As a result, the normalized weights β1-β3 indicate a proportion of the raw weights α1-α3 relative to the first sum αsum of the raw weights.
As illustrated in
Using the weighted sum technique illustrated in
with si[n] the ith directional output (e.g., beam from the beamformer, although the disclosure is not limited thereto) and βi a corresponding weight.
As the weights βi vary over time, a more accurate representation is:
where βr,i[n] is the recursive weight (e.g., weight smoothed over time) for the ith directional output at a current time index n, βr,i[n−1] is the recursive weight for the ith directional output at a previous time index n−1, ρ1 is a first smoothing parameter (e.g., smoothing value), βi is an instantaneous weight (e.g., weight calculated above using Equation 4) for the current time index n, sr[n] is the recursive combined output (e.g., combined output smoothed over time) at the current time index n, sr[n−1] is the recursive combined output at the previous time index n−1, ρ2 is a second smoothing parameter, and si[n] the ith directional output. Thus, instead of determining the combined output s[n] based on the instantaneous weights (e.g., βi) and instantaneous directional outputs (e.g., si[n]) as illustrated in Equation 9, the device 102 may determine a recursive combined output sr[n] and recursive weight βr,i by factoring in previous time indexes.
Thus, to improve a quality of the second audio output 128b, the device 102 may control a rate of growth and/or a rate of decay of the second audio output 128b (e.g., recursive combined output sr[n]) using recursive weights βr,i. A smaller smoothing parameter (e.g., ρ1 or ρ2 closer to 0) provides less smoothing and prioritizes the instantaneous values of the weights and directional outputs si[n], whereas a larger smoothing parameter (e.g., ρ1 or ρ2 closer to 1) provides greater smoothing and prioritizes previous values of the recursive weights βr,i[n] and previous values of the recursive combined output sr[n]. As the recursive weights βr,i[n] and recursive combined output sr[n] are determined recursively, the first smoothing parameter ρ1 and the second smoothing parameter ρ2 may be determined based on a desired number of iterations and therefore a duration of time. For example, if the device 102 wants to smooth the recursive weights βr,i[n] and recursive combined output sr[n] using values over the past 100 ms, the device 102 may use a smoothing parameter value of ρ=0.9. If the device 102 wants to smooth the recursive weights βr,i[n] and recursive combined output sr[n] using values over the past 200 ms, the device 102 may use a smoothing parameter value of ρ=0.95.
As described above, the weights βi may be determined based on a magnitude of power or SNR value corresponding to a directional output. However, changing the weights too rapidly may result in artifacts or other degradations of the combined output Sout. To improve a quality of the combined output Sout, the device 102 may control a rate of change of a weight b1 by only increasing or decreasing a raw weight αi up to a fixed percentage. For example, to control a rate of change when a power (e.g., pi) of a directional output Si is decreasing, the device 102 may select a decay parameter value λ that is a positive constant less than one (e.g., 0.98) and prevent the raw weight αi from decreasing by more than the decay parameter value λ. Additionally or alternatively, the device 102 may use the decay parameter value λ to have a fixed rate of change when the raw weight αi is decreasing. For example, if the power pi of a directional output Si is decreasing (e.g., pi[n]<pi[n−1]), the device 102 may determine the raw weight αi using the decay parameter value λ (e.g., αi=λ αi[n−1]). Thus, the raw weight αi (and therefore the normalized weight bi) may increase without restriction but may only decrease at a constant rate, as shown in equation 11:
While the example illustrated above and equation 11 controls the rate of decay, the disclosure is not limited thereto and the device 102 may control the rate of decay and/or a rate of growth without departing from the disclosure.
In contrast, the device 102 may control a rate of decay by limiting a rate of change for the raw weight αi when the when the power pi is decreasing. As illustrated in
The device 102 may determine (712) signal metrics for the directional outputs 124, such as signal to noise ratios (SNRs) or a magnitude of power associated with the directional outputs 124. For example, the device 102 may determine a first SNR associated with the first directional output 124a, a second SNR associated with a second directional output 124b, etc. However, SNR is just a single example of a parameter or characteristic associated with the directional outputs 124 and the present disclosure is not limited thereto. Instead of using SNRs, the device 102 may compare the directional outputs 124 using other characteristics or parameters (e.g., signal strength, power, etc.).
The device 102 may determine (714) a single directional output corresponding to a highest signal metric and may select (716) the single directional output as a first audio output 128a. For example, the device 102 may determine that the second SNR associated with the second directional output 124b is the largest SNR value of the directional outputs 124 and may select the second directional output 124b as the first audio output 128a.
The device 102 may determine (718) a sum of the signal metrics. For example, the device 102 may add the first SNR, the second SNR and so on until the mth SNR to generate a sum (e.g., αsum). In some examples, the device 102 may multiply the signal metrics by a variable (e.g., γi), as discussed above.
The device 102 may optionally determine (720) a difference in signal metrics and/or weights. For example, the device 102 may perform the technique illustrated above with regard to
The device 102 may generate (724) a second audio output 128b by combining the directional outputs 124 using the weights. For example, the device 102 may generate the second audio output 128b by multiplying the directional outputs 124 by corresponding weights (e.g., first directional output 124a multiplied by the first weight, second directional output 124b multiplied by the second weight, etc.) and summing the products, as discussed above with regard to equations 3 and/or 9.
In the equal weighting technique, the device 102 may attempt to preserve a loudness of each talker when generating the second audio output 128b. Thus, instead of determining individual weights based on corresponding signal metrics, the device 102 may select a first group of directional outputs to have a first weight and a second group of directional outputs to have a second weight. As a result, the second audio output 128b will preserve a loudness of each of the first group of directional inputs (relative to each other) while reducing a loudness of ambient noise and reverberation.
To perform the equal weighting technique, the device 102 may determine signal metrics (e.g., SNR values) for individual directional outputs 124 and separate the directional outputs 124 into groups based on the signal metrics. For example, the device 102 may separate the directional outputs 124 into the first group (e.g., SET1) and the second group (e.g., SET2) using a fixed threshold. Thus, directional outputs 124 having a signal metric above the fixed threshold are associated with the first group and directional outputs 124 having a signal metric below the fixed threshold are associated with the second group. The first group may collectively get a first total weight wtot1 (e.g., wtot1=0.7) and the second group may collectively get a second total weight wtot2 (e.g., wtot2=0.3), with the first total weight wtot1 and the second total weight wtot2 adding up to a value of one (e.g., wtot1 wtot2=1). If there are P directional outputs 124 in the first group and Q directional outputs 124 in the second group (for a total number of directional outputs 124 equal to P+Q), the device 102 may determine a first individual weight wind1 of the first group as:
and a second weight of the second group as:
Thus, if there are six directional outputs (e.g., 124a-124f) and the device 102 associates three directional outputs (e.g., 124a-124c) with the first group SET1 having a first total weight wtot1 of 0.7 and three directional outputs (e.g., 124d-124f) with the second group SET2 having a second total weight wtot2 of 0.3, the first individual weight w1 may be equal to 0.233 and the second individual weight wind2 may be equal to 0.1.
As discussed above, a weight w1 for an ith directional output 124 may be determined over time. Thus, while the first directional output 124a is included in the first group, a corresponding weight w1 may be equal to the first individual weight wind1 (e.g., w1(n)=wd1). However, when the first directional output 124a changes to the second group, the first weight w1 may be equal to the second individual weight wind2 (e.g., w1(n+1)=wind2). As discussed above with regard to
si[n]=ρ3si[n−1]+(1−ρ3)wi[n] [13]
In some examples, the device 102 may select the third smoothing parameter ρ3 to be a slow time constant ρslow when the directional output changed sets within the last K frames (e.g., directional output changes from the second group to the first group or vice versa) and a normal time constant ρnorm otherwise, where the slow time constant ρslow is larger than the normal time constant ρnorm (e.g., ρslow<ρnorm).
A smaller time constant (e.g., third smoothing parameter ρ3 closer to 0) provides less smoothing and prioritizes the instantaneous values of the weights wi[n], whereas a larger time constant (e.g., third smoothing parameter ρ3 closer to 1) provides greater smoothing and prioritizes previous values of the directional output si. As the directional output si is determined recursively, the third smoothing parameter ρ3 may be determined based on a desired number of iterations and therefore a duration of time. For example, if the device 102 wants to smooth the directional output si using values over the past 100 ms, the device 102 may use a third smoothing parameter value of ρ3=0.9. If the device 102 wants to smooth the directional output si using values over the past 200 ms, the device 102 may use a third smoothing parameter value of ρ3=0.95. Therefore, when the directional output has not changed sets within the last K frames, the device may select the normal time constant ρnorm to be a first value (e.g., 0.9) that corresponds to a first period of time (e.g., 100 ms), whereas when the directional output has changed sets within the last K frames, the device 102 may select the slow time constant ρslow to be a larger value (e.g., 0.95), which corresponds to a longer period of time (e.g., 200 ms) in order to lessen the impact of the instantaneous values of the weights wi[n].
In some examples, the device 102 may generate a first output Sout1 using a sum of the directional outputs 124 and output the first output Sout1 as the second audio output 128b:
Sout1[n]=Σsi[n] [14]
However, in other examples the second audio output 128b may smooth the first output Sout1 to generate a second output Sout2 to output as the second audio output 128b:
Sout2[n]=ρ4Sout2[n−1]+(1−ρ4)Sout1[n] [15]
where ρ4 is a fourth smoothing parameter between zero and one. Thus, a smaller smoothing parameter (e.g., ρ4 closer to 0) provides less smoothing and prioritizes the instantaneous values of the first output Sout1, whereas a larger smoothing parameter (e.g., ρ4 closer to 1) provides greater smoothing and prioritizes previous values of the second output Sout2. As the second output Sout2 is determined recursively, the fourth smoothing parameter ρ4 may be determined based on a desired number of iterations and therefore a duration of time. For example, if the device 102 wants to smooth the second output Sout2 using values over the past 100 ms, the device 102 may use a smoothing parameter of ρ4=0.9. If the device 102 wants to smooth the second output Sout2 using values over the past 200 ms, the device 102 may use a smoothing parameter of ρ4=0.95.
After grouping the directional outputs S1-S6, the device 102 may assign a first weight (β1) to the first group and a second weight (β2) to the second group, as illustrated by Weights 814. The first group may have a first total weight βtot1 (e.g., 0.7) while the second group may have a second total weight βtot2 (e.g., 0.3). Thus, in this example, the first weight (β1) may be approximately 0.233 and the second weight (β2) may be approximately 0.1.
The device 102 may multiply the first directional output 124a (e.g., S1) by the first weight β1 to generate a first product. multiply the second directional output 124b (e.g., S2) by the first weight β1 to generate a second product. and multiply the fourth directional output 124d (e.g., S4) by the first weight β1 to generate a third product. Similarly, the device 102 may multiply the third directional output 124c (e.g., S3) by the second weight β2 to generate a fourth product, multiply the fifth directional output 124e (e.g., S5) by the second weight β2 to generate a fifth product. and multiply the sixth directional output 124f (e.g., S6) by the second weight β2 to generate a sixth product. The device 102 may determine the combined output (e.g., Sout) by summing portions of the directional outputs S1-S6 (e.g., products), as illustrated by beam combiner 816.
While
Additionally or alternatively, the device 102 may use a 3/3 configuration 920 and select three directional outputs (e.g., 124a, 124b and 124d) in the first group and the remaining three directional outputs (e.g., 124c, 124e and 124f) in the second group, as shown by group chart 922. As the first total weight βtot1 and the second total weight βtot2 remain fixed, the first weight β1 is decreased (e.g., β1=0.7/3=0.233) and the second weight is increased (e.g., β2=0.3/3=0.1).
Additionally or alternatively, the device 102 may use a 4/2 configuration 930 and select four directional outputs (e.g., 124a, 124b, 124d and 124e) in the first group and the remaining two directional outputs (e.g., 124c and 124f) in the second group, as shown by group chart 932. As the first total weight βtot1 and the second total weight βtot2 remain fixed, the first weight β1 is decreased (e.g., β1=0.7/4=0.175) and the second weight is increased (e.g., β2=0.3/2=0.15).
The examples illustrated in
After grouping the directional outputs S1-S6, the device 102 may assign a first weight (β1) to the first group and a second weight (β2) to the second group, as illustrated by Weights 1014. The first group may have a first total weight βtot1 (e.g., 0.7) while the second group may have a second total weight βtot2 (e.g., 0.3). Thus, in this example, the first weight (β1) may be approximately 0.233 and the second weight (β2) may be approximately 0.1.
The device 102 may multiply the first directional output 124a (e.g., S1) by the first weight β1 to generate a first product. multiply the second directional output 124b (e.g., S2) by the first weight β1 to generate a second product. and multiply the fourth directional output 124d (e.g., S4) by the first weight Pi to generate a third product. Similarly, the device 102 may multiply the third directional output 124c (e.g., S3) by the second weight β2 to generate a fourth product, multiply the fifth directional output 124e (e.g., S5) by the second weight β2 to generate a fifth product. and multiply the sixth directional output 124f (e.g., S6) by the second weight β2 to generate a sixth product. The device 102 may determine the combined output (e.g., Sout) by summing portions of the directional outputs S1-S6 (e.g., products), as illustrated by beam combiner 1016.
Additionally or alternatively, the device 102 may use a 2/2/2 configuration 1020 and select two directional outputs (e.g., 124a and 124d) in a first group, two directional outputs (e.g., 124b and 124e) in a second group and the remaining two directional outputs (e.g., 124c, and 124f) in a third group, as shown by group chart 1022.
After grouping the directional outputs S1-S6, the device 102 may assign a first weight (β1) to the first group, a second weight (β2) to the second group and a third weight (β3) to the third group, as illustrated by Weights 1024. The first group may have a first total weight βtot1 (e.g., 0.6), the second group may have a second total weight βtot2 (e.g., 0.3) and the third group may have a third total weight βtot3 (e.g., 0.1). Thus, in this example, the first weight (β1) may be approximately 0.3, the second weight (β2) may be approximately 0.15 and the third weight (β3) may be approximately 0.05.
The device 102 may multiply the first directional output 124a (e.g., S1) by the first weight β1 to generate a first product and multiply the fourth directional output 124d (e.g., S4) by the first weight β1 to generate a second product. Similarly, the device 102 may multiply the second directional output 124b (e.g., S2) by the second weight β2 to generate a third product and multiply the fifth directional output 124e (e.g., S5) by the second weight β2 to generate a fourth product. Finally, the device 102 may multiply the third directional output 124c (e.g., S3) by the third weight β3 to generate a fifth product and multiply the sixth directional output 124f (e.g., S6) by the third weight β3 to generate a sixth product. The device 102 may determine the combined output (e.g., Sout) by summing portions of the directional outputs S1-S6 (e.g., products), as illustrated by beam combiner 1026.
The examples illustrated in
As illustrated in
In contrast, the second technique (e.g., static threshold 1120) may select a variable number of directional outputs in each group based on a static threshold 1120. For example, signal metric chart 1122 illustrates the static threshold 1120 between the second signal metric (e.g., α2) and the fifth signal metric (e.g., α5). Thus, group chart 1124 illustrates that the device 102 selects the signal metrics (e.g., α1, α2 and α4) above the static threshold 1120 as a first group and the signal metrics (e.g., α3, α5 and α6) below the static threshold 1120 as a second group. Depending on a value of the static threshold 1120, the device 102 may select additional directional outputs in the first group and/or second group. For example, if the static threshold 1120 was higher, the device 102 may only select the first signal metric (e.g., α1) and the fourth signal metric (e.g., α4) in the first group, including the second signal metric (e.g., α2) in the second group.
While the static threshold 1120 may separate directional outputs using a fixed threshold, the static threshold 1120 may not intelligently separate the directional outputs based on relative signal metrics of the directional outputs. In contrast, the third technique (e.g., dynamic threshold 1130) may select a variable number of directional outputs in each group based on a dynamic threshold 1130 that takes into account differences between signal metrics. For example, the device 102 may determine a first grouping of signal metrics (e.g., α1, α2, α4 and α5) within a threshold of each other, a second grouping of signal metrics (e.g., α3 and α6) and that there is a gap between the first grouping and the second grouping, as illustrated in signal metric chart 1132. Based on the groupings and the gap between the groupings, the device 102 may select signal metrics (e.g., α1, α2, α4 and α5) as a first group and signal metrics (e.g., α3 and α6) as a second group. While the example illustrated in
While
The device 102 may determine (1212) signal metrics for the directional outputs 124, such as signal to noise ratios (SNRs) or a magnitude of power associated with the directional outputs 124. For example, the device 102 may determine a first SNR associated with the first directional output 124a, a second SNR associated with a second directional output 124b, etc. However, SNR is just a single example of a parameter or characteristic associated with the directional outputs 124 and the present disclosure is not limited thereto. Instead of using SNRs, the device 102 may compare the directional outputs 124 using other characteristics or parameters (e.g., signal strength, power, etc.).
The device 102 may determine (1214) a single directional output corresponding to a highest signal metric and may select (1216) the single directional output as a first audio output 128a. For example, the device 102 may determine that the second SNR associated with the second directional output 124b is the largest SNR value of the directional outputs 124 and may select the second directional output 124b as the first audio output 128a.
The device 102 may determine (1218) a threshold, select (1220) a first plurality of directional outputs with signal metrics above the threshold and select (1222) a second plurality of directional outputs with signal metrics below the threshold. For example, the device 102 may use the techniques discussed above with regard to
The device 102 may assign (1224) a first weight to the first plurality of directional outputs and may assign (1226) a second weight to the second plurality of directional outputs. For example, the device 102 may determine a first total weight and a second total weight, determine the first weight using the first total weight and a first number of directional outputs in the first plurality, and determine the second weight using the second total weight and a second number of directional outputs in the second plurality, as discussed in greater detail above with regard to
While steps 1220-1226 illustrate an example of the device 102 separating the directional outputs into two different groups, the disclosure is not limited thereto and the device 102 may separate the directional outputs into any number of groups without departing from the disclosure. For example, the device 102 may select a 3/3 configuration (e.g., two groups), a 2/2/2 configuration (e.g., three groups) or the like, as illustrated in
The device 102 may generate (1230) a second audio output 128b by combining the directional outputs 124 using the weights. For example, the device 102 may generate the second audio output 128b by multiplying the directional outputs 124 by corresponding weights (e.g., first directional output 124a multiplied by a first weight, second directional output 124b multiplied by a second weight, etc.) and summing the products.
The system 100 may include one or more audio capture device(s), such as a microphone 118 or an array of microphones 118. The audio capture device(s) may be integrated into the device 102 or may be separate.
The system 100 may also include an audio output device for producing sound, such as speaker(s) 1316. The audio output device may be integrated into the device 102 or may be separate.
The device 102 may include an address/data bus 1124 for conveying data among components of the device 102. Each component within the device 102 may also be directly connected to other components in addition to (or instead of) being connected to other components across the bus 1124.
The device 102 may include one or more controllers/processors 1304, that may each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory 1306 for storing data and instructions. The memory 1306 may include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive (MRAIVI) and/or other types of memory. The device 102 may also include a data storage component 1308, for storing data and controller/processor-executable instructions (e.g., instructions to perform the algorithms illustrated in
Computer instructions for operating the device 102 and its various components may be executed by the controller(s)/processor(s) 1304, using the memory 1306 as temporary “working” storage at runtime. The computer instructions may be stored in a non-transitory manner in non-volatile memory 1306, storage 1308, or an external device. Alternatively, some or all of the executable instructions may be embedded in hardware or firmware in addition to or instead of software.
The device 102 includes input/output device interfaces 1302. A variety of components may be connected through the input/output device interfaces 1302, such as the speaker(s) 1316, the microphones 118, and a media source such as a digital media player (not illustrated). The input/output interfaces 1302 may include A/D converters for converting the output of microphone 118 into echo signals y 120, if the microphones 118 are integrated with or hardwired directly to device 102. If the microphones 118 are independent, the A/D converters will be included with the microphones 118, and may be clocked independent of the clocking of the device 102. Likewise, the input/output interfaces 1302 may include D/A converters for converting the reference signals x 112 into an analog current to drive the speakers 114, if the speakers 114 are integrated with or hardwired to the device 102. However, if the speakers 114 are independent, the D/A converters will be included with the speakers 114, and may be clocked independent of the clocking of the device 102 (e.g., conventional Bluetooth speakers).
The input/output device interfaces 1302 may also include an interface for an external peripheral device connection such as universal serial bus (USB), FireWire, Thunderbolt or other connection protocol. The input/output device interfaces 1302 may also include a connection to one or more networks 1399 via an Ethernet port, a wireless local area network (WLAN) (such as WiFi) radio, Bluetooth, and/or wireless network radio, such as a radio capable of communication with a wireless communication network such as a Long Term Evolution (LTE) network, WiMAX network, 3G network, etc. Through the network 1399, the system 100 may be distributed across a networked environment.
The device 102 further includes an adaptive beamformer 104, an acoustic echo canceller (AEC) 108 and an output generator 126. The adaptive beamformer 104 includes a fixed beamformer (FBF) 105, a multiple input canceler (MC) 106 and a blocking matrix (BM) 107. The AEC 108 may include one or more discrete logic blocks used to perform acoustic echo cancellation.
Multiple devices 102 may be employed in a single system 100. In such a multi-device system, each of the devices 102 may include different components for performing different aspects of the AEC process. The multiple devices 102 may include overlapping components. The components of device 102 as illustrated in
The concepts disclosed herein may be applied within a number of different devices and computer systems, including, for example, general-purpose computing systems, multimedia set-top boxes, televisions, stereos, radios, server-client computing systems, telephone computing systems, laptop computers, cellular phones, personal digital assistants (PDAs), tablet computers, wearable computing devices (watches, glasses, etc.), other mobile devices, etc.
The above aspects of the present disclosure are meant to be illustrative. They were chosen to explain the principles and application of the disclosure and are not intended to be exhaustive or to limit the disclosure. Many modifications and variations of the disclosed aspects may be apparent to those of skill in the art. Persons having ordinary skill in the field of digital signal processing and echo cancellation should recognize that components and process steps described herein may be interchangeable with other components or steps, or combinations of components or steps, and still achieve the benefits and advantages of the present disclosure. Moreover, it should be apparent to one skilled in the art, that the disclosure may be practiced without some or all of the specific details and steps disclosed herein.
Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture such as a memory device or non-transitory computer readable storage medium. The computer readable storage medium may be readable by a computer and may comprise instructions for causing a computer or other device to perform processes described in the present disclosure. The computer readable storage medium may be implemented by a volatile computer memory, non-volatile computer memory, hard drive, solid-state memory, flash drive, removable disk and/or other media. Some or all of the Acoustic Echo Canceller 108 may be implemented by a digital signal processor (DSP).
As used in this disclosure, the term “a” or “one” may include one or more items unless specifically stated otherwise. Further, the phrase “based on” is intended to mean “based at least in part on” unless specifically stated otherwise.